- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work shows that a deep SSM with recursion over a small number of blocks can match the performance of multiple independently parameterized blocks. In addition, it introduces a reshaping operation that aggregates multiple scalar inputs into chunks, aiming to maintain performance while improving efficiency. Overall, however, this work lacks novelty, and both its methodology and experimental setup are not sufficiently rigorous.
Critical strengths
Overall, this work aims to provide some insights through the analysis of experimental phenomena. At least from this motivation, it is worth encouraging.
Critical weaknesses
First, the idea of block recurrence has long appeared in many classical designs, such as Neural ODEs and Deep Equilibrium Models. Reusing this idea in SSMs is straightforward, yet it offers no new understanding, either theoretically or practically. For instance, the conclusion of the proposition is too trivial to warrant standing as a separate proposition.
Second, the so-called "consistent time-granularity" merely aggregates sequence inputs from different time steps. A major concern is whether this introduces information leakage in strict time-series problems, unless each prediction is also constructed at such a block level. Moreover, "time" in SSMs typically refers to the in their modeling.
Finally, the empirical study and analysis are far from sufficient. The initial and most notable advantage of SSMs lies in their ability to follow complex dynamics and capture long-range dependencies. However, this work does not emphasize such experiments, especially long-sequence modeling benchmarks like PathfinderX in the Long Range Arena. In addition, the experiments lack necessary case analyses, leaving readers unclear about where exactly the proposed approach contributes.
Questions for authors
Please refer to the Weakness part.
4 comments
Many thanks for the authors' response.
Regarding Response 1, what does "systematic" refer to: diversity in dimensions, or coverage of diverse Deep State-Space Model (DSSM) classes? If it refers to diversity in investigation dimensions, the deep recursion and so-called time-granularity consistency explored by the authors appear insufficient. First, these two perspectives are relatively trivial and do not convey any unexpected conclusions. Second, the temporal granularity discussed here is chunk-level granularity rather than the temporal discretization step size of SSMs, whereas the latter is of greater concern in DSSM research. If it refers to the diversity of model classes, DSSM research has developed many typical designs that can serve as anchors—such as S4, Liquid-S4, S5, DSS, GSS, Mamba, Mamba2, H3, LRU, and Jamba, among others. Discussing only LRU, S5, LinOSS, and LrcSSM seems insufficiently systematic. At the very least, it should cover the various distinct design philosophies among them.
Regarding Response 3, long-sequence modeling capability was a key feature of the DSSM family of models when they were initially developed, a characteristic primarily demonstrated by the performance differences on PathfinderX. S4 was also the first model to succeed on this task, whereas previous attention-based approaches failed on it, even though they can work on tasks like EigenWorms and PPG-DaLiA [1, 2]. Therefore, long-sequence modeling capability is a major advantage of the DSSM family, and the proposed design philosophy should not compromise this essential capability. As a benchmark to validate long-sequence modeling capability, PathfinderX is evidently more persuasive than EigenWorms and PPG-DaLiA, this stems from a difference in task difficulty rather than merely sequence length. On the other hand, the mentioned "case study" does not refer to component ablation, but rather using concrete instance cases to help readers understand precisely where the proposed approach takes effect, what performance improvements or drops it brings, or demonstrating this clearly through synthetic data. For example, does the stacked model exhibit a diminished capability in tracking non-stationary long sequences? Alternatively, how does utilizing the so-called consistent time-granularity affect the tracking capability for variable-period long sequences? Questions of this nature can help readers gain a deeper understanding of the practical behavioral differences that these designs introduce to DSSMs, thereby providing deeper insight into the logic underlying the surface-level experimental results.
[1] Moreno-Pino F, Arroyo Á, Waldon H, et al. Rough transformers: Lightweight and continuous time series modelling through signature patching[J]. Advances in Neural Information Processing Systems, 2024, 37: 106264-106294.
[2] Kataria S, Wu Y, Chen Z, et al. Generalist vs Specialist Time Series Foundation Models: Investigating Potential Emergent Behaviors in Assessing Human Health Using PPG Signals[J]. arXiv preprint arXiv:2510.14254, 2025.
Regarding Response 1, what does "systematic" refer to...
By systematic, we refer to the controlled evaluation of depth recurrence and input reshaping across different SSM formulations, not an exhaustive coverage of all existing DSSM architectures:
For a fixed six-layer architecture, we investigate all possible complete periodic depth-sharing configurations (AAAAAA, ABABAB, ABCABC, and the independent ABCDEF baseline), considering both final-layer and block-wise supervision. We apply the same experimental framework across four SSM architectures representing different modeling principles: LRU and S5 (linear dynamics with different parameterizations), LinOSS (oscillatory dynamics), and LrcSSM (selective, nonlinear dynamics). This allows us to investigate whether the observed effects are specific to a particular state-space formulation or extend across different underlying dynamics.
We agree that additional architectures would further broaden the scope of our findings. Mamba-based model experiments are in the Appendix. We believe that the four selected models in the main text already provide meaningful diversity in their underlying dynamical formulations.
Furthermore, while depth-wise parameter sharing may appear conceptually straightforward, its empirical results are not trivial. Our results show that depth-recurrent models remain competitive with independently parameterized baselines despite having substantially fewer unique parameters. Across the 144 evaluated depth-recurrent configurations, 132 (91.7%) achieve either a higher mean performance or performance within one standard deviation of the corresponding independent baseline.
Regarding temporal granularity, we agree with the Reviewer that our investigation concerns input reshaping and chunk-level granularity, and not the numerical discretization step size of SSMs. These are two distinct domains. Our aim in the paper was to investigate how changing the input representation by modifying the information density (consequently the sequence length) affects performance.
Regarding Response 3, long-sequence modeling capability...
We thank the Reviewer for further clarifying these concerns. Regarding long-sequence modeling, we agree that PathfinderX is a challenging benchmark for evaluating long-range dependencies. However, we believe that EigenWorms and PPG-DaLiA remain meaningful evaluations of long-sequence modeling on real-world time-series data. In particular, EigenWorms contains sequences of 17,984 time steps, exceeding the length of PathfinderX.
We would also like to clarify a point regarding reference [1]. They report that Transformer gets OOM errors on the full EigenWorms dataset, even when 70% of the input observations are removed. While their proposed Rough Transformer successfully addresses this limitation, the findings indicate that EigenWorms also presents substantial challenges for standard attention-based architectures.
More importantly, our work investigates depth-wise recurrence rather than introducing a new temporal recurrence mechanism. Sharing parameters across depth does not truncate the input sequence or alter the temporal recurrence within individual SSM blocks. Our objective is to investigate how these architectural modifications affect the performance of existing SSMs compared with their independently parameterized counterparts. Our experimental results provide evidence that depth recurrence can preserve competitive performance on the evaluated tasks, despite substantially reducing the number of unique parameters.
Our current experiments focus on comparing these architectural choices under consistent experimental conditions across different SSM formulations and datasets. In particular, the depth-sharing configurations allow us to assess the trade-off between parameter efficiency and predictive performance, while the input-reshaping experiments investigate the effect of changing the information representation presented to the model.
Thanks for the authors' response.
I believe the response addresses my first follow-up question. However, regarding the second question on PathfinderX, it is an important benchmark that distinguishes DSSM architectures from earlier attention-based ones. Even though EigenWorms and PPG-DaLiA have longer sequence lengths, they are clearly less challenging than PathfinderX, which, as I noted earlier, stems from differences in task difficulty rather than sequence length alone. For example, tracking a very long periodic oscillation is straightforward, whereas tracking an aperiodic impulse signal is much harder. Moreover, the claim that "EigenWorms also poses significant challenges to standard attention-based architectures" arises because quadratic complexity limits the direct application of vanilla attention on it, yet related work such as Rough Transformer can still tackle this task with an attention-based architecture, whereas PathfinderX cannot.
Finally, I understand that this paper does not aim to introduce a new mechanism, but rather to investigate depth recurrence in DSSMs. For this exact reason, the requested case studies are necessary, because current experiments make it hard to see how depth recurrence actually helps improve DSSM performance beyond simply showing that adding depth recurrence allows DSSMs to reduce parameters while boosting performance. More importantly, understanding the underlying reasons and logic helps readers grasp the behavioral mechanisms behind this phenomenon. This is what an exploratory paper should achieve: presenting a phenomenon together with deeper mechanistic insights, rather than merely describing the phenomenon itself.
We thank the Reviewer for their feedback.
We thank the Reviewer for this observation. We agree that depth recurrence and parameter sharing are well-established concepts. Our contribution is not the introduction of depth recurrence itself, but its systematic investigation across four SSM architectures with different underlying dynamics, considering multiple parameter-sharing configurations and supervision strategies.
Our experiments provide practical insights into parameter efficiency, showing that depth-recurrent SSMs can achieve comparable or better performance with substantially fewer parameters. Furthermore, our parameter-matched ablations (Appendix A.5) and adaptive inference experiments demonstrate benefits beyond parameter reduction alone.
Regarding the proposition, we agree that the result follows directly from parameter sharing. Its purpose is to clarify that the observed improvements arise despite a restricted hypothesis class, and not from increased expressivity. We will make this motivation clearer in the manuscript.
We thank the Reviewer for raising this concern. Our experiments primarily involve sequence-level classification, where the complete input sequence is available before prediction. Therefore, grouping observations from different time steps does not introduce information leakage. The same applies to sequence-to-one regression when predictions are made after observing the complete input window.
For strictly causal online tasks, grouping consecutive observations requires buffering, potentially introducing latency or reducing the prediction frequency. We already acknowledge this limitation in Section 4.5 and will clarify it further.
We also agree that "consistent time-granularity" may be misleading, as our approach modifies the input representation rather than the temporal discretization of the underlying SSM. We will revise that terminology to input concentration.
We thank the Reviewer for this suggestion. While our work primarily investigates parameter efficiency and input representation rather than introducing new mechanisms for long-range memory, our experiments already include long sequences, such as EigenWorms (17,984 steps) and PPG-DaLiA (49,920 steps), both exceeding PathfinderX (16,384 steps).
Regarding the experimental analysis, we would like to highlight several analyses already included in the paper. First, we systematically compare depth-recurrence configurations across four SSM architectures and six classification and regression benchmarks. Second, our parameter-matched ablation in Appendix A.5 shows that the observed improvements cannot be attributed solely to parameter reduction. Third, our adaptive inference experiments investigate the trade-off between predictive performance and computational cost, demonstrating that fewer block applications can maintain comparable performance while reducing GFLOPs. Finally, our input reshaping experiments examine how the effectiveness of the proposed transformation varies with the dimensionality of the input data.
These experiments provide insights into both the advantages and limitations of the proposed design choices. In particular, we observe that depth recurrence is most consistently beneficial for LRU, whereas the improvements are more heterogeneous for LinOSS and LrcSSM. We find that input reshaping provides the clearest benefits for datasets with high-dimensional observations, while its effects are smaller for medium-dimensional inputs. We will emphasize these findings more clearly.