acceptodds
Under review as a conference paper at ICLR 2027

Reshape and Recur: Improving SSMs with Input Reshaping and Depth Recurrence

Abstract

State Space Models (SSMs) are increasingly deployed in the Edge because they offer, at comparable performance, a smaller memory/training/inference footprint, compared to Large Language Models (LLMs). These three advantages are a direct consequence of the time recurrence inherent in the SSMs architecture. Here, we further improve this recurrent architecture by positively answering two previously underexplored, orthogonal questions: (1) Can we reduce SSMs memory-footprint without any performance penalty, by also employing depth recurrence? (2) Can we increase SSMs performance by using a fixed and consistent time-granularity across all tasks? The first question is somewhat unexpected, given that SSMs are already recurrent. However, the orthogonal depth recurrence further decreases SSMs memory footprint. We show that a looped SSM with parameters adaptively iterated times, achieves a performance comparable to a standard SSM with independent parameters, where . The second question is also unexpected given the time-recurrent nature of the SSMs architecture. However, it makes perfect sense for the time-parallel training of SSMs on the entire input sequence. We show that concatenating time steps for lower-dimensional sequence elements, or flattening and re-chunking the joint feature-time dimension for high-dimensional ones, can improve the baseline by enhancing the way information is presented to the model. Our results for both extensions lead to consistent benefits across four representative SSM architectures: LRU, S5, LinOSS, LrcSSM.

Primary area:
learning on time series and dynamical systems
Keywords:
State Space Models, Depth-Recurrence, Input Reshaping, Efficient Sequence Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RNueOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work shows that a deep SSM with recursion over a small number of blocks can match the performance of multiple independently parameterized blocks. In addition, it introduces a reshaping operation that aggregates multiple scalar inputs into chunks, aiming to maintain performance while improving efficiency. Overall, however, this work lacks novelty, and both its methodology and experimental setup are not sufficiently rigorous.

Critical strengths

Overall, this work aims to provide some insights through the analysis of experimental phenomena. At least from this motivation, it is worth encouraging.

Critical weaknesses

First, the idea of block recurrence has long appeared in many classical designs, such as Neural ODEs and Deep Equilibrium Models. Reusing this idea in SSMs is straightforward, yet it offers no new understanding, either theoretically or practically. For instance, the conclusion of the proposition is too trivial to warrant standing as a separate proposition.

Second, the so-called "consistent time-granularity" merely aggregates sequence inputs from different time steps. A major concern is whether this introduces information leakage in strict time-series problems, unless each prediction is also constructed at such a block level. Moreover, "time" in SSMs typically refers to the Δt\Delta t in their modeling.

Finally, the empirical study and analysis are far from sufficient. The initial and most notable advantage of SSMs lies in their ability to follow complex dynamics and capture long-range dependencies. However, this work does not emphasize such experiments, especially long-sequence modeling benchmarks like PathfinderX in the Long Range Arena. In addition, the experiments lack necessary case analyses, leaving readers unclear about where exactly the proposed approach contributes.

Questions for authors

Please refer to the Weakness part.

4 comments

We thank the Reviewer for their feedback.

First, the idea of block recurrence has long appeared in many classical designs, such as Neural ODEs and Deep Equilibrium Models. Reusing this idea in SSMs is straightforward, yet it offers no new understanding, either theoretically or practically. For instance, the conclusion of the proposition is too trivial to warrant standing as a separate proposition.

We thank the Reviewer for this observation. We agree that depth recurrence and parameter sharing are well-established concepts. Our contribution is not the introduction of depth recurrence itself, but its systematic investigation across four SSM architectures with different underlying dynamics, considering multiple parameter-sharing configurations and supervision strategies.

Our experiments provide practical insights into parameter efficiency, showing that depth-recurrent SSMs can achieve comparable or better performance with substantially fewer parameters. Furthermore, our parameter-matched ablations (Appendix A.5) and adaptive inference experiments demonstrate benefits beyond parameter reduction alone.

Regarding the proposition, we agree that the result follows directly from parameter sharing. Its purpose is to clarify that the observed improvements arise despite a restricted hypothesis class, and not from increased expressivity. We will make this motivation clearer in the manuscript.

Second, the so-called "consistent time-granularity" merely aggregates sequence inputs from different time steps. A major concern is whether this introduces information leakage in strict time-series problems, unless each prediction is also constructed at such a block level. Moreover, "time" in SSMs typically refers to the δt\delta t in their modeling.

We thank the Reviewer for raising this concern. Our experiments primarily involve sequence-level classification, where the complete input sequence is available before prediction. Therefore, grouping observations from different time steps does not introduce information leakage. The same applies to sequence-to-one regression when predictions are made after observing the complete input window.

For strictly causal online tasks, grouping consecutive observations requires buffering, potentially introducing latency or reducing the prediction frequency. We already acknowledge this limitation in Section 4.5 and will clarify it further.

We also agree that "consistent time-granularity" may be misleading, as our approach modifies the input representation rather than the temporal discretization of the underlying SSM. We will revise that terminology to input concentration.

Finally, the empirical study and analysis are far from sufficient. The initial and most notable advantage of SSMs lies in their ability to follow complex dynamics and capture long-range dependencies. However, this work does not emphasize such experiments, especially long-sequence modeling benchmarks like PathfinderX in the Long Range Arena. In addition, the experiments lack necessary case analyses, leaving readers unclear about where exactly the proposed approach contributes.

We thank the Reviewer for this suggestion. While our work primarily investigates parameter efficiency and input representation rather than introducing new mechanisms for long-range memory, our experiments already include long sequences, such as EigenWorms (17,984 steps) and PPG-DaLiA (49,920 steps), both exceeding PathfinderX (16,384 steps).

Regarding the experimental analysis, we would like to highlight several analyses already included in the paper. First, we systematically compare depth-recurrence configurations across four SSM architectures and six classification and regression benchmarks. Second, our parameter-matched ablation in Appendix A.5 shows that the observed improvements cannot be attributed solely to parameter reduction. Third, our adaptive inference experiments investigate the trade-off between predictive performance and computational cost, demonstrating that fewer block applications can maintain comparable performance while reducing GFLOPs. Finally, our input reshaping experiments examine how the effectiveness of the proposed transformation varies with the dimensionality of the input data.

These experiments provide insights into both the advantages and limitations of the proposed design choices. In particular, we observe that depth recurrence is most consistently beneficial for LRU, whereas the improvements are more heterogeneous for LinOSS and LrcSSM. We find that input reshaping provides the clearest benefits for datasets with high-dimensional observations, while its effects are smaller for medium-dimensional inputs. We will emphasize these findings more clearly.

Many thanks for the authors' response.

Regarding Response 1, what does "systematic" refer to: diversity in dimensions, or coverage of diverse Deep State-Space Model (DSSM) classes? If it refers to diversity in investigation dimensions, the deep recursion and so-called time-granularity consistency explored by the authors appear insufficient. First, these two perspectives are relatively trivial and do not convey any unexpected conclusions. Second, the temporal granularity discussed here is chunk-level granularity rather than the temporal discretization step size of SSMs, whereas the latter is of greater concern in DSSM research. If it refers to the diversity of model classes, DSSM research has developed many typical designs that can serve as anchors—such as S4, Liquid-S4, S5, DSS, GSS, Mamba, Mamba2, H3, LRU, and Jamba, among others. Discussing only LRU, S5, LinOSS, and LrcSSM seems insufficiently systematic. At the very least, it should cover the various distinct design philosophies among them.

Regarding Response 3, long-sequence modeling capability was a key feature of the DSSM family of models when they were initially developed, a characteristic primarily demonstrated by the performance differences on PathfinderX. S4 was also the first model to succeed on this task, whereas previous attention-based approaches failed on it, even though they can work on tasks like EigenWorms and PPG-DaLiA [1, 2]. Therefore, long-sequence modeling capability is a major advantage of the DSSM family, and the proposed design philosophy should not compromise this essential capability. As a benchmark to validate long-sequence modeling capability, PathfinderX is evidently more persuasive than EigenWorms and PPG-DaLiA, this stems from a difference in task difficulty rather than merely sequence length. On the other hand, the mentioned "case study" does not refer to component ablation, but rather using concrete instance cases to help readers understand precisely where the proposed approach takes effect, what performance improvements or drops it brings, or demonstrating this clearly through synthetic data. For example, does the stacked model exhibit a diminished capability in tracking non-stationary long sequences? Alternatively, how does utilizing the so-called consistent time-granularity affect the tracking capability for variable-period long sequences? Questions of this nature can help readers gain a deeper understanding of the practical behavioral differences that these designs introduce to DSSMs, thereby providing deeper insight into the logic underlying the surface-level experimental results.

[1] Moreno-Pino F, Arroyo Á, Waldon H, et al. Rough transformers: Lightweight and continuous time series modelling through signature patching[J]. Advances in Neural Information Processing Systems, 2024, 37: 106264-106294.

[2] Kataria S, Wu Y, Chen Z, et al. Generalist vs Specialist Time Series Foundation Models: Investigating Potential Emergent Behaviors in Assessing Human Health Using PPG Signals[J]. arXiv preprint arXiv:2510.14254, 2025.

Regarding Response 1, what does "systematic" refer to...

By systematic, we refer to the controlled evaluation of depth recurrence and input reshaping across different SSM formulations, not an exhaustive coverage of all existing DSSM architectures:

For a fixed six-layer architecture, we investigate all possible complete periodic depth-sharing configurations (AAAAAA, ABABAB, ABCABC, and the independent ABCDEF baseline), considering both final-layer and block-wise supervision. We apply the same experimental framework across four SSM architectures representing different modeling principles: LRU and S5 (linear dynamics with different parameterizations), LinOSS (oscillatory dynamics), and LrcSSM (selective, nonlinear dynamics). This allows us to investigate whether the observed effects are specific to a particular state-space formulation or extend across different underlying dynamics.

We agree that additional architectures would further broaden the scope of our findings. Mamba-based model experiments are in the Appendix. We believe that the four selected models in the main text already provide meaningful diversity in their underlying dynamical formulations.

Furthermore, while depth-wise parameter sharing may appear conceptually straightforward, its empirical results are not trivial. Our results show that depth-recurrent models remain competitive with independently parameterized baselines despite having substantially fewer unique parameters. Across the 144 evaluated depth-recurrent configurations, 132 (91.7%) achieve either a higher mean performance or performance within one standard deviation of the corresponding independent baseline.

Regarding temporal granularity, we agree with the Reviewer that our investigation concerns input reshaping and chunk-level granularity, and not the numerical discretization step size of SSMs. These are two distinct domains. Our aim in the paper was to investigate how changing the input representation by modifying the information density (consequently the sequence length) affects performance.

Regarding Response 3, long-sequence modeling capability...

We thank the Reviewer for further clarifying these concerns. Regarding long-sequence modeling, we agree that PathfinderX is a challenging benchmark for evaluating long-range dependencies. However, we believe that EigenWorms and PPG-DaLiA remain meaningful evaluations of long-sequence modeling on real-world time-series data. In particular, EigenWorms contains sequences of 17,984 time steps, exceeding the length of PathfinderX.

We would also like to clarify a point regarding reference [1]. They report that Transformer gets OOM errors on the full EigenWorms dataset, even when 70% of the input observations are removed. While their proposed Rough Transformer successfully addresses this limitation, the findings indicate that EigenWorms also presents substantial challenges for standard attention-based architectures.

More importantly, our work investigates depth-wise recurrence rather than introducing a new temporal recurrence mechanism. Sharing parameters across depth does not truncate the input sequence or alter the temporal recurrence within individual SSM blocks. Our objective is to investigate how these architectural modifications affect the performance of existing SSMs compared with their independently parameterized counterparts. Our experimental results provide evidence that depth recurrence can preserve competitive performance on the evaluated tasks, despite substantially reducing the number of unique parameters.

Our current experiments focus on comparing these architectural choices under consistent experimental conditions across different SSM formulations and datasets. In particular, the depth-sharing configurations allow us to assess the trade-off between parameter efficiency and predictive performance, while the input-reshaping experiments investigate the effect of changing the information representation presented to the model.

Thanks for the authors' response.

I believe the response addresses my first follow-up question. However, regarding the second question on PathfinderX, it is an important benchmark that distinguishes DSSM architectures from earlier attention-based ones. Even though EigenWorms and PPG-DaLiA have longer sequence lengths, they are clearly less challenging than PathfinderX, which, as I noted earlier, stems from differences in task difficulty rather than sequence length alone. For example, tracking a very long periodic oscillation is straightforward, whereas tracking an aperiodic impulse signal is much harder. Moreover, the claim that "EigenWorms also poses significant challenges to standard attention-based architectures" arises because quadratic complexity limits the direct application of vanilla attention on it, yet related work such as Rough Transformer can still tackle this task with an attention-based architecture, whereas PathfinderX cannot.

Finally, I understand that this paper does not aim to introduce a new mechanism, but rather to investigate depth recurrence in DSSMs. For this exact reason, the requested case studies are necessary, because current experiments make it hard to see how depth recurrence actually helps improve DSSM performance beyond simply showing that adding depth recurrence allows DSSMs to reduce parameters while boosting performance. More importantly, understanding the underlying reasons and logic helps readers grasp the behavioral mechanisms behind this phenomenon. This is what an exploratory paper should achieve: presenting a phenomenon together with deeper mechanistic insights, rather than merely describing the phenomenon itself.

Reviewer_f6poOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to study the periodic expressivity of SSM in terms of time and depth recurrence. The authors first study the expressivity of SSM with depth recurrence and how it improve SSM memory-footprint. The authors then investigate how sequence reshape (time recurrence) impacts model performance. Evaluations on multiple SSM model families show how the depth recurrence improve memory footprint while time recurrence shows limited improvement to model performance.

Critical strengths
  1. Simple modification to SSM with practical benefits. It is intuitive that through depth and time recurrence, one can reduce the memory footprint and improve the compute parallelization of the model.
  2. Breadth within the experimental scope. Multiple SSM families are evaluated on both classification and regression tasks.
  3. Interesting theoretical contribution to the expressivity of the depth recurrence that in turn motivates the research question
Critical weaknesses
  1. The "without any performance penalty" framing is too strong. We can observe in table 2 that there is non-negligible accuracy reduction between the independent baseline and the shared configuration
  2. Depth recurrance translates to parameter saving while inference time memory footprint like KV-caching is not attributed clearly in writing. Also there is optimization memory cost for train time which should also be taken in account. The paper should separate these types of memory footprints.
  3. Novelty needs narrower positioning. Although the paper cites [1], its discussion emphasizes their implicit fixed-point formulation without adequately addressing their explicitly unrolled and weight-tied Mamba2 models. Furthermore, [2] already uses temporal patching before, contradicting the broad characterization that prior SSMs process one original timestep per input token.

[1] M. Schone et al. Implicit Language Models are RNNs: Balancing Parallelization and Expressivity [2] H. Ma et al. A Mamba Foundation Model for Time Series Forecasting

Questions for authors

Please see the weaknesses

2 comments

We thank the Reviewer for their feedback.

The "without any performance penalty" framing is too strong. We can observe in table 2 that there is non-negligible accuracy reduction between the independent baseline and the shared configuration

We thank the Reviewer for this observation. We will revise "without any performance penalty", as some configurations exhibit reduced accuracy compared with the independently parameterized baseline. However, as shown in Table 2, the fully shared configuration matches or exceeds the baseline mean in 19 out of 24 settings. More broadly, across all 144 depth-recurrent configurations, 132 (91.7%) either achieve a higher mean performance or remain within one standard deviation of the baseline mean.

These results show that depth recurrence generally preserves competitive predictive performance while substantially reducing the number of trainable parameters.

Depth recurrance translates to parameter saving while inference time memory footprint like KV-caching is not attributed clearly in writing. Also there is optimization memory cost for train time which should also be taken in account. The paper should separate these types of memory footprints.

First, we would like to point out that unlike attention-based Transformers, the SSM architectures considered in our work do not rely on KV caching, but instead maintain recurrent states during sequential inference. Therefore, KV caching is not applicable to our setup.

Our primary contribution is about parameter memory, which is reduced through weight sharing across depth. As shown in Figure 2 and Table 3, the fully shared configuration achieves a 5.3-5.9x reduction in parameter count. This also reduces the memory required for parameter gradients and optimizer states during training.

Moreover, Table 4 already distinguishes parameter count, FLOPs, latency, and peak memory. At fixed depth, peak memory remains approximately unchanged despite parameter reductions. However, adaptive inference with two block applications reduces FLOPs to 0.35x, latency to 0.33x, and peak memory to 0.55x relative to the independent baseline. Figure 4 further demonstrates the computational savings achievable through adaptive inference.

We will highlight these distinctions more clearly in the main text.

Novelty needs narrower positioning. Although the paper cites [1], its discussion emphasizes their implicit fixed-point formulation without adequately addressing their explicitly unrolled and weight-tied Mamba2 models. Furthermore, [2] already uses temporal patching before, contradicting the broad characterization that prior SSMs process one original timestep per input token. [1] M. Schone et al. Implicit Language Models are RNNs: Balancing Parallelization and Expressivity [2] H. Ma et al. A Mamba Foundation Model for Time Series Forecasting

We thank the Reviewer for highlighting these connections. We agree that Schöne et al. [1] investigate both implicit fixed-point models and explicitly unrolled, weight-tied Mamba2 architectures. Similarly, Ma et al. [2] employ temporal patching in SSM-based forecasting. We will expand our related work discussion to acknowledge both contributions more explicitly.

Our work still differs in its systematic evaluation of depth recurrence across four distinct SSM architectures, considering multiple parameter-sharing patterns, supervision strategies, and adaptive inference.

Furthermore, while Ma et al. employ learned convolutional embeddings of temporal patches, our input reshaping directly redistributes the input across the feature and sequence dimensions without introducing additional parameters in the reshaping operation itself.

Our contribution is the systematic investigation these design choices and their effects on predictive performance and parameter efficiency.

I appreciate the authors for their clarification. However, as the novelty of the paper remains limited to the systematic evaluations of the designs, I think the submission is more suited for other venues. I'd encourage the author to improve the framing and the contribution of the work while taking broader literature in this area into account.

Reviewer_5hgwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates two complementary modifications to State Space Models (SSMs):

Reusing SSM blocks across layers rather than independently parameterizing each layer, with optional intermediate supervision and adaptive inference based on prediction stability.

Concatenating adjacent timesteps for low-dimensional inputs or flattening and rechunking high-dimensional inputs to change the effective sequence length and feature granularity.

Experiments across four SSM architectures and six time-series benchmarks show that fully shared models use 5.3–5.9× fewer parameters while frequently maintaining comparable accuracy. Input reshaping improves performance on selected datasets, and adaptive inference reduces the number of layer applications, often from six to two or three.

Critical strengths
  1. Convincing evidence for parameter-efficient depth recurrence.

The paper systematically evaluates parameter sharing across four distinct SSM architectures, including parameter-matched shallow baselines (Appendix A.5). Repeated blocks can outperform equally parameterized shallow models while approaching the performance of deeper independent models. This supports the practical value of iterative computation beyond simple parameter reduction. The substantial reduction in model weights is particularly relevant for memory-constrained deployment.

  1. Simple, complementary interventions with useful empirical ablations.

Depth recurrence and input reshaping can be implemented without redesigning the underlying SSM dynamics. The authors evaluate them separately and jointly, investigate different supervision strategies, and demonstrate inference-time computation savings through adaptive stopping. The approach is therefore relatively easy to reproduce and potentially applicable across existing architectures.

Critical weaknesses
  1. Input reshaping is confounded with substantially different computational budgets.

For high-dimensional inputs, reshaping from 61–63 features to width 8 increases the effective sequence length by approximately 7.6–7.9×. Consequently, the SSM performs substantially more sequential state updates, while also processing features in an artificial order. Without compute-matched comparisons against conventional patching, learned feature projections, or alternative feature orderings, it is unclear whether the gains arise from the proposed representation principle or simply additional recurrent computation.

Moreover, Table 4 reports approximately unchanged FLOPs, latency, and peak memory under fixed-depth recurrence despite the large parameter reduction. This distinction weakens the broader efficiency claims.

  1. Limited evidence for generality and insufficient differentiation from related work.

Depth iteration in SSMs has substantial precedent in Schöne et al. (2025), while Deng et al. (2026, LT2) and Yu et al. (2026, Looped State-Space Language Models/CHASE) investigate explicit looping of Mamba-based architectures at larger scales. Temporal patching is also well established through PatchTST. The distinctive contribution therefore appears to be the particular empirical combination and systematic comparison, rather than a fundamentally new architectural principle.

The main evaluation uses only six-layer models with roughly 25K–155K parameters and five UEA classification datasets. Furthermore, several improvements are small relative to their reported standard deviations, and the paper treats results within one standard deviation as evidence of comparable performance, which is not a formal equivalence test. These limitations make the claimed consistency and scalability difficult to assess.

Questions for authors
  1. Are the gains from input reshaping simply due to more computation? Reshaping increases sequence length by nearly 8× on some datasets. Can you compare against standard input representations at the same FLOPs? Does changing feature order affect accuracy?
  2. How reliable are the reported improvements? Why use custom train/test splits instead of the official UEA splits? How was data leakage prevented in PPG-DaLiA? Are the improvements statistically significant?
  3. Does adaptive stopping actually improve efficiency? At the same computational cost, does adaptive stopping outperform simply running a fixed number of fewer iterations? How was the stopping threshold chosen?
  4. What is new compared to previous work on looped SSMs? How does your approach differ from Schöne et al. (2025), Deng et al. (2026), and Yu et al. (2026)? What contribution goes beyond applying existing depth-recurrence techniques to additional SSM architectures?
2 comments

We thank the Reviewer for their constructive feedback.

Input reshaping is confounded with substantially different computational budgets.

We agree that, for high-dimensional inputs, reshaping increases the effective sequence length. Our goal with input reshaping is not necessarily to reduce computation, but to investigate how changing the information concentration per model step affects predictive performance while keeping the underlying SSM architecture fixed.

As shown in Figure 5 and Table 5, reshaping improves the performance of several architectures, including LRU on Heartbeat (71.3% to 75.5%) and S5 on MotorImagery (54.4% to 61.8%).

Following the Reviewer's suggestion, we evaluated LRU under compute-matched configurations. The results below show that the input-reshaping improvement is not solely due to additional recurrent computation but to better representations.

DatasetOriginal sizeCompute-matched setup with conventional input sizeInput-reshaped (ours)
Heartbeat71.29 ± 1.2172.26 ± 6.8075.48 ± 2.37
Motor56.14 ± 6.3753.68 ± 4.0958.60 ± 6.04

Moreover, Table 4 reports approximately unchanged FLOPs, latency, and peak memory under fixed-depth recurrence despite the large parameter reduction. This distinction weakens the broader efficiency claims.

Regarding efficiency, we would like to clarify that our primary claim for fixed-depth recurrence is about parameter efficiency and not reduced computational cost. As shown in Figure 2 and Table 3, the fully shared configuration reduces the number of parameters by 5.3-5.9x. Table 4 explicitly reports that FLOPs, latency, and peak memory remain approximately unchanged at fixed depth, since the number of block applications is preserved.

However, computational savings arise through adaptive inference, where executing only two block applications reduces FLOPs to 0.35x, latency to 0.33x, and peak memory to 0.55x relative to the independent baseline.

The main evaluation uses only six-layer models with roughly 25K–155K parameters and five UEA classification datasets...

Our experimental setup deliberately fixes the depth to six layers and the state and hidden dimensions to 64 across all four SSM architectures. This controlled evaluation allows us to isolate the effects of depth recurrence and input reshaping without confounding them with architecture-specific hyperparameter choices. We explain this in Section 4.2 and Appendix A.2.2.

Although our experiments focus on relatively small models, they cover four architectures with distinct state dynamics and six classification and regression benchmarks with sequence lengths ranging from 405 to 49,920. Extending these experiments to larger models could provide additional evidence of scalability, but we believe the current results already establish the practical value of these design choices under controlled conditions.

We agree that being within one standard deviation of the baseline does not necessarily mean that the models perform equally well. We use this comparison to account for variability across the five random seeds and to indicate comparable performance.

  1. Are the gains from input reshaping simply due to more computation? Reshaping increases sequence length by nearly 8× on some datasets. Can you compare against standard input representations at the same FLOPs? Does changing feature order affect accuracy?

We thank the Reviewer for these suggestions. We agree that the increased sequence length may introduce additional computational cost, making it important to distinguish the benefits of input reshaping from those of increased computation. See our experimental results above.

Regarding feature ordering, our method applies a deterministic reshaping operation that preserves all input information and does not require any particular feature ordering. While different feature permutations may affect performance, investigating their influence is beyond the primary scope of our work, which focuses on the effects of input dimensionality and sequence length on SSM performance.

(Response cont.)

  1. How reliable are the reported improvements? Why use custom train/test splits instead of the official UEA splits? How was data leakage prevented in PPG-DaLiA? Are the improvements statistically significant?

We use fixed 70/15/15 train/validation/test splits with predefined random seeds, consistently across all architectures and configurations. These were established in prior sequence modeling work of [1,2,3, etc.]. Regarding PPG-DaLiA, we follow the data preprocessing and splitting protocol of the existing benchmark implementation on which our experiments are based. Our motivation was to follow these setups and to ensure controlled comparisons under identical experimental conditions.

[1] Benjamin Walker, Andrew D. McLeod, Tiexin Qin, Yichuan Cheng, Haoliang Li, Terry Lyons. Log Neural Controlled Differential Equations: The Lie Brackets Make a Difference. [2] T. Konstantin Rusch, Daniela Rus. Oscillatory State-Space Models. ICLR 2025. [3] Monika Farsang, Radu Grosu. Parallelization of Non-linear State-Space Models: Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling. NeurIPS 2025.

  1. Does adaptive stopping actually improve efficiency? At the same computational cost, does adaptive stopping outperform simply running a fixed number of fewer iterations? How was the stopping threshold chosen?

Our adaptive inference experiments (Figure 4 and Table 8) demonstrate that the same trained model can execute a variable number of block applications, typically 2-5 instead of the six used during training, while maintaining competitive performance and reducing computational cost. In contrast, using a fixed number of iterations may either reduce predictive performance or result in unnecessary computation, depending on the input. The stopping threshold was selected using the validation set.

  1. What is new compared to previous work on looped SSMs? How does your approach differ from Schöne et al. (2025), Deng et al. (2026), and Yu et al. (2026)? What contribution goes beyond applying existing depth-recurrence techniques to additional SSM architectures?

We thank the Reviewer for highlighting these related works. We agree that depth recurrence and parameter sharing are established concepts, including the explicitly unrolled Mamba2 models investigated by Schöne et al. (2025).

However, we would like to clarify the publication timeline. The first version of our work was made publicly available on arXiv on May 15, 2026, preceding Deng et al. (May 20, 2026) and Yu et al. (July 11, 2026). To preserve anonymity during the double-blind review process, we kindly ask the Reviewer to accept the correctness of this timeline without attempting to verify the identity of the submission.

Furthermore, we would like to point out that these works are currently available as arXiv preprints. According to the ICLR 2027 Reviewer Guidelines, "authors are encouraged, but not required, to compare against non-peer-reviewed arXiv papers, and the absence of such comparisons should not be grounds for rejection". Nevertheless, we acknowledge their relevance and will expand our related work discussion accordingly.

Our contribution lies in the systematic investigation of depth recurrence across four SSM architectures with different underlying dynamics, including multiple parameter-sharing patterns, supervision strategies, parameter-matched ablations, and adaptive early stopping. We additionally investigate input reshaping as a complementary design choice. We will clarify these distinctions and further acknowledge related work in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.