acceptodds
Under review as a conference paper at ICLR 2027

When Expressivity Does not Help: A Study of Linear and Nonlinear RNNs for Time Series Forecasting

Abstract

Time series forecasting requires models to capture temporal dependencies across diverse time scales and dynamics. Recent work has explored more expressive recurrent architectures for forecasting, including mechanisms with input-dependent transitions and explicit state-tracking capabilities. Several studies have reported performance improvements over simpler recurrent models. However, whether these gains systematically arise from greater recurrence expressivity, and whether capabilities emphasized in general sequence modeling are broadly relevant to forecasting, remains unclear. We present a systematic empirical study of recurrent models with different levels of recurrence expressivity for time series forecasting. Overall, by using bootstrap hypothesis testing, we find that greater recurrence expressivity does not significantly improve forecasting accuracy. On controlled synthetic forecasting tasks, models with nonlinearity or explicit state-tracking mechanisms do not show a systematic advantage over less expressive architectures. Furthermore, on standard long-term forecasting and diverse real-world benchmarks, LTSF and GIFT-Eval, competitive linear models such as mLSTM can match or outperform several nonlinear architectures. Moreover, within linear-recurrence models, added expressivity through input-dependent transitions or explicit state-tracking mechanisms does not reliably improve performance. Our results suggest that recurrence expressivity alone is not a reliable predictor of forecasting performance, highlighting the importance of dataset characteristics and architectural inductive biases beyond expressivity.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, SSMs, RNNs, expressivity, state tracking, recurrent neural networks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SZAeOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work presents an empirical comparison for time series forecasting across 12 linear and nonlinear architectures, categorized across four axes defined by the authors. The comparison is conducted across both synthetic and real-life benchmarks, and the main takeaways are that more expressivity in an architecture - measured by state tracking capacity - does not present measurable or significant gains on time series forecasting.

Critical strengths
  • Overall experimental setup is solid. The equal-shot hyperparameter search, the multi-seed experiments, the synthetic and real-life benchmark evals and the OLS regression analysis show a stable and rigorous evaluation protocol.
  • The study covers multiple families of linear and nonlinear RNNs.
Critical weaknesses
  • I can accept that nonlinear recurrences or 'more expressivity' are not required for good forecasting performance, however the paper lacks an explanation on which aspects do matter and influence the forecasting performance. Right now the paper's message is that there is no clear correlation between the examined structural aspects of the architectures and the forecasting performance. An insight as to what does matter would be important for a paper like this.

  • My second point is connected to the first one. I think reporting negative results or "certain architectural choices seem to not correlate with forecasting performance" is a good contribution to the field, but only if it provides insights that are valuable to the community. There are large gaps in performance between models classified in the exact same category according to the 4 criteria devised by the authors. Examples of different average ranks for models that appear in the same model class:

    • Cross-Variable synthetic results avg rank: RNN: 1.88, LSTM: 8.5, sLSTM: 9.62, GRU: 6.25.
    • LTSF avg. rank: RNN: 5.33, LSTM: 8.78, sLSTM: 3.00, GRU: 4.78.
    • Trend synthetic results avg rank: mLSTM: 6.59, Mamba-2: 8.45, Mamba: 2.64.
    • LTSF avg. rank: mLSTM: 2.56, Mamba: 8.56, Mamba-2: 8.22.
  • I accept the argument that tasks generally associated with state tracking (parity, permutations etc.) are not relevant to time series forecasting, however: The ranking in the synthetic tasks are at best loosely correlated to real-life benchmark performance. This is also apparent from the above 4 comparisons. This could indicate that the chosen synthetic tasks - while relevant to time series analysis - may not exploit the state tracking capacity the model architectures present.

  • The composite synthetic task, along with Appendix H's table: the authors argue that mLSTM and sLSTM, as well as DeltaNet and Gated DeltaProduct (state tracking counterparts of each other) are 'close' in performance, however it appears to me that both mLSTM and DeltaNet are better than their state tracking counterpart. Moreover, the negative eigenvalues in DeltaNet seem to lower loss only very marginally, but then gated deltaproduct increases it. These are noisy results and are inconclusive to me.

Few more minor points I noticed:

  • Griffin does not appear in the table describing the architectures but suddenly appear in the comparison.
  • I appreciate the rigorous hyperparam search, however disclosing the final hyperparameters of the models would be important, as it can contain important information about the models. Now the paper's findings are less reproducible.
Questions for authors
  • Mamba is best overall on the synthetics (5.00) and near-worst on LTSF (8.56); S5 is best on GIFT and is at best in the middle on LTSF. Which, if any, of the synthetic categories do the authors consider predictive of real-world performance, and on what evidence?
  • The current takeaways from the paper is that expressivity does not help. The results on the different benchmarks do not reveal a clear ordering for me, can the authors establish more clean takeaways, with actionable insights on architecture choices for designing more efficient/better performing time series forecasting models?
  • mLSTM and Mamba-2 are identical rows in Table 1, yet rank 2.56 vs 8.22 on LTSF. What do the authors attribute this to? can it be due to gating/normalization, the matrix state, tuning budget? If it's something the taxonomy doesn't capture, shouldn't that factor be an axis?
5 comments

We thank the reviewer for recognizing the rigor of our experimental evaluation, including the equal-budget hyperparameter search, multi-seed experiments, diverse synthetic and real-world benchmarks, and statistical analysis. We also appreciate the reviewer’s acknowledgment of the breadth of recurrent architectures covered in our study.

Weakness 1:

I can accept that nonlinear recurrences or 'more expressivity' are not required for good forecasting performance, however the paper lacks an explanation on which aspects do matter and influence the forecasting performance. Right now the paper's message is that there is no clear correlation between the examined structural aspects of the architectures and the forecasting performance. An insight as to what does matter would be important for a paper like this.

We thank the reviewer for raising this important point. This point has also been addressed in our response to Weakness 2 from Reviewer djGF. We agree that identifying the factors that influence forecasting performance, beyond recurrence expressivity, is an important aspect of understanding the empirical results. To address this, we have added a Discussion section (Section 5) to highlight three observations from our experiments and the hypotheses we made based on them.

First, the choice of inductive bias appears to matter for robustness to noise. In particular, on the forecasting tasks with synthetic noisy datasets with periodicity, fixed-transition models such as S5 and LRU, as well as the vanilla RNN, perform particularly well under high noise. This suggests that simpler, time-invariant state transitions may be advantageous when forecasting signals with strong cyclic structure in noisy observations.

Second, certain architectural design choices may benefit forecasting performance independently of recurrence expressivity. In particular, we observe strong performance from both mLSTM and sLSTM on several real-world datasets. Despite their differences in recurrence expressivity, these models share exponential gating and normalizer states, which we hypothesize may contribute to their strong performance.

Finally, context length emerges as an important factor beyond architectural choices. Our statistical analysis identifies it as a significant factor influencing forecasting performance, suggesting that tuning the context length deserves particular attention during model selection and hyperparameter optimization. (New Table 21 in the paper)

We have incorporated these observations into the new Discussion section, while emphasizing that the proposed architectural explanations are hypotheses that motivate further investigation.

Weakness 2:

My second point is connected to the first one. I think reporting negative results or "certain architectural choices seem to not correlate with forecasting performance" is a good contribution to the field, but only if it provides insights that are valuable to the community. There are large gaps in performance between models classified in the exact same category according to the 4 criteria devised by the authors. Examples of different average ranks for models that appear in the same model class:

  • Cross-Variable synthetic results avg rank: RNN: 1.88, LSTM: 8.5, sLSTM: 9.62, GRU: 6.25.
  • LTSF avg. rank: RNN: 5.33, LSTM: 8.78, sLSTM: 3.00, GRU: 4.78.
  • Trend synthetic results avg rank: mLSTM: 6.59, Mamba-2: 8.45, Mamba: 2.64.
  • LTSF avg. rank: mLSTM: 2.56, Mamba: 8.56, Mamba-2: 8.22.

We thank the reviewer for emphasizing the importance of providing insights that are valuable to the community. We believe that our findings have practical implications for the design of recurrent forecasting architectures. In particular, they suggest that increasing recurrence expressivity, for example by moving beyond diagonal state transitions, may not be necessary to achieve strong forecasting performance. This motivates the use of simpler and computationally less expensive architectures for time series forecasting. For example, diagonal SSMs such as Mamba are considerably more computationally affordable than models with non-diagonal state transitions, such as DeltaProduct. Since moving beyond diagonal transitions is partly motivated by enabling more expressive state-tracking capabilities, our results suggest that competitive forecasting models, including potentially more efficient time series foundation models, could be developed using diagonal SSMs. We have expanded the Discussion section to elaborate on these implications.

Regarding the performance differences between models within the same expressivity category, we agree that these are important observations. As we also discuss in our response to Weakness 1, architectures within the same category can differ in other design choices that are not captured by our four expressivity criteria. For example, although LSTM and sLSTM belong to the same expressivity category, sLSTM introduces exponential gating and normalizer states, which distinguish its state-update mechanism from that of a standard LSTM without changing its state-tracking expressivity under our criteria. Interestingly, these design choices are shared by mLSTM, which also exhibits strong forecasting performance despite belonging to a different expressivity category. This observation motivates our hypothesis that architectural properties beyond recurrence expressivity, such as exponential gating and state normalization, may play an important role in forecasting performance. We have added this observation to the Discussion section and identified a systematic investigation of these architectural properties as an important direction for future work.

We would also like to emphasize that this difference in models ranking in performance has already been observed for time series forecasting on other types of architectures, for example, [Brigato et al., 2026] also reports different best-performing models on different datasets.


[Brigato et al., 2026]: Lorenzo Brigato, Rafael Morand, Knut Strømmen, Maria Panagiotou, Markus Schmidt, Stavroula Mougiakakou, There are no Champions in Supervised Long-Term Time Series Forecasting

Weakness 3:

I accept the argument that tasks generally associated with state tracking (parity, permutations etc.) are not relevant to time series forecasting, however: The ranking in the synthetic tasks are at best loosely correlated to real-life benchmark performance. This is also apparent from the above 4 comparisons. This could indicate that the chosen synthetic tasks - while relevant to time series analysis - may not exploit the state tracking capacity the model architectures present.

We agree that model rankings vary between synthetic and real-world datasets (again also in accordance with earlier observations from [Brigato et al., 2026]), and even across the two real-world benchmarks. However, an important observation is that our main finding remains consistent: models with simpler recurrence mechanisms and lower expressivity remain among the strongest performers across these settings. We would also like to clarify that, when comparing synthetic and real-world results, the overall ranking across synthetic datasets may provide a more representative picture than the ranking on any individual synthetic dataset, such as the Cross-Variable dataset highlighted in Weakness 2.

Nevertheless, we find the reviewer's point important. The fact that performance rankings do not consistently transfer from synthetic to real-world datasets raises the question of whether existing synthetic forecasting tasks adequately capture the properties of real-world time series, and, in particular, whether they sufficiently exploit the state-tracking capabilities of recurrent architectures. We see this as an underexplored question in the time series forecasting literature. Indeed, whether state-tracking capabilities are necessary for forecasting, and whether the synthetic tasks commonly used in time series research require such capabilities, were among the questions motivating our study. Although tasks such as parity and permutation tracking are generally considered canonical state-tracking problems, their relevance to forecasting is unclear. Our experiments suggest that the state-tracking capabilities studied in the literature may not translate into consistent forecasting benefits on the synthetic and real-world datasets we considered. This is particularly relevant given that several recent recurrent forecasting studies have motivated more expressive architectures partly through their state-tracking capabilities.

We agree that the synthetic tasks considered in our study may not fully exploit these capabilities. However, this possibility does not undermine our findings: rather, it highlights that the forecasting tasks currently used in empirical evaluations, including commonly used synthetic datasets, may not require the additional state-tracking expressivity offered by more sophisticated recurrent architectures. Investigating more structured synthetic processes, for example those generated by chaotic dynamical systems, and examining whether their forecasting benefits from more expressive state-tracking mechanisms, is an interesting direction for future work.

Weakness 4:

The composite synthetic task, along with Appendix H's table: the authors argue that mLSTM and sLSTM, as well as DeltaNet and Gated DeltaProduct (state tracking counterparts of each other) are 'close' in performance, however it appears to me that both mLSTM and DeltaNet are better than their state tracking counterpart. Moreover, the negative eigenvalues in DeltaNet seem to lower loss only very marginally, but then gated deltaproduct increases it. These are noisy results and are inconclusive to me.

We thank the reviewer for pointing this out. We agree that the results do not consistently support the claim that these model pairs perform comparably. For the composite synthetic task (Table 3), looking at test MSE rather than average rankings provides a more complete picture, as the performance advantage is relatively evenly distributed between mLSTM and sLSTM, and similarly between DeltaNet and DeltaProduct. We therefore consider the ranking metric complementary to the test MSE when interpreting these results. However, we acknowledge that the results in Appendix G (the old Appendix H) show a clearer preference for the less expressive counterparts, i.e., DeltaNet and negative-Deltanet over DeltaProduct. We will revise our description to reflect this asymmetry rather than characterize their performance as comparable.

We would also note that the stronger forecasting performance of DeltaNet compared to DeltaProduct is consistent with observations in recent literature, including [Fu et al. (2026)]. This provides additional context for our observation that the more expressive DeltaProduct architecture does not necessarily offer a forecasting advantage over DeltaNet.


[Fu et al., 2026]: X Fu, Y Li, G Papaioannou, Y Kim, Reverso: Efficient Time Series Foundation Models for Zero-shot Forecasting

Reviewer_2CjFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a systematic empirical study of 12 linear and nonlinear recurrent architectures for time series forecasting. Motivated by recent claims that formal recurrence expressivity and state-tracking capabilities are crucial for forecasting, the authors benchmark these models across controlled synthetic tasks (SynTSBench) and real-world datasets (LTSF and GIFT-Eval). Using bootstrap OLS regression and rank comparisons, the study finds that higher recurrence expressivity and state tracking do not systematically improve forecasting accuracy, with simple time-invariant linear SSMs (like S5 and LRU) often matching or outperforming more expressive counterparts.

Critical strengths
  1. Questioning whether formal language-modeling properties like state tracking (e.g., parity or FSA simulation) actually matter for continuous time series forecasting is a well-motivated research direction, and the head-to-head comparison within model families (such as mLSTM vs. sLSTM and DeltaNet vs. DeltaProduct) provides clear evidence that extra transition complexity rarely helps.

  2. Evaluating models in controlled synthetic settings on SynTSBench (under varying noise levels and lags) offers clean insights into why simple linear SSMs like S5 and LRU can recover underlying signals more reliably than more complex recurrent models.

Critical weaknesses
  1. All 12 recurrent models are tested in a basic point-wise setup without patching, where they generally trail standard baselines like PatchTST. Since modern time-series models typically apply recurrence over patch-level tokens, it is unclear whether these findings fully carry over to patch-based architectures. This is the primary concern about this paper.

  2. On the real-world benchmarks, hyperparameters were tuned on one representative dataset per group and reused across other datasets in that group. Since recurrent architectures can be sensitive to learning rates and state sizes, this reuse might smooth out some genuine differences between models.

  3. In the regression analysis (Table 2), several architectural features overlap across the 12 models (for example, all nonlinear models in Table 1 are also non-diagonal and state-tracking), which leads to wide confidence intervals and makes it harder to isolate the impact of each individual factor.

Questions for authors

no more questions

4 comments

We thank the reviewer for recognizing the importance and relevance of our research question, as well as the value of our controlled synthetic experiments in providing insights into the effects of different recurrent architectural properties on forecasting performance.

Weakness 1:

All 12 recurrent models are tested in a basic point-wise setup without patching, where they generally trail standard baselines like PatchTST. Since modern time-series models typically apply recurrence over patch-level tokens, it is unclear whether these findings fully carry over to patch-based architectures. This is the primary concern about this paper.

We thank the reviewer for raising this important point. We agree that patching is a potentially important design choice and that investigating whether our findings extend to patch-based recurrent architectures is valuable.

To investigate the effect of patching, we added experiments in Appendix D (Table 15), where we rerun the LTSF experiments using patch-based inputs, following the PatchTST design with a patch length of 16 and a stride of 8, followed by an encoder MLP and the recurrent model. For this comparison, we retain the hyperparameters selected in the original point-wise experiments, rather than performing a separate hyperparameter search for the patched setting.

Table 16 in the same appendix shows the results for datasets per models, averaged over horizons. The main observation here is that the final ranking of the models does not change drastically from what we had in the non-patched experiments on LTSF, especially the best performing models remain mLSTM and sLSTM.

However, we acknowledge that patching does lower the error for several models. Of the 22 dataset–horizon settings in that table, the patched test MSE is lower for mLSTM in 18, for DeltaProduct in 16, for LRU in 16, for Mamba in 15, for DeltaNet in 14, and for S5 in 13. On the other hand, LSTM is lower in 6 of these 22 settings and higher in 15. sLSTM is lower in 8 and higher in 14. RNN, GRU, Griffin, and M2RNN go in both directions.

Another observation is that, as could be expected, the effect of patching is not uniform across models and datasets. For example, On Electricity at horizon 96, S5 goes from 0.435 to 0.260. The same change raises the error elsewhere. On Electricity at horizon 720, S5 goes from 0.778 to 0.970. On ETTh1 at horizon 96, S5 and mLSTM fall while GRU and LSTM rise.

A sample of more remarkable changes from non-patched to patched setup is shown in the following table:

Table (i) Percent change in test MSE from the unpatched results to Table 15. Negative means a lower patched error.

DatasetHGRULSTMS5MambamLSTMDeltaNet
ETTh196+16%+25%-13%-2%-15%-16%
Weather96-12%+9%+9%+3%-2%+14%
Electricity96+3%-5%-40%-6%-5%+1%
Electricity720-57%+34%+25%-35%-6%+19%

Looking across all shared dataset–horizon configurations, several models, including mLSTM, DeltaProduct, LRU, Mamba, Mamba-2, S5, and DeltaNet, achieve lower test MSE more frequently with patching, whereas LSTM more frequently exhibits higher errors. The remaining models show mixed effects. Overall, these results support the conclusion that forecasting performance depends on the interaction between architectural choices and the characteristics of the data, and that neither greater recurrence expressivity nor patching alone guarantees better performance.

Still, Weakness 1

Finally, we would like to add that there are, however, a few considerations behind our original point-wise setup.

First, unlike in attention-based architectures, where patching has become a widely adopted design choice (e.g., PatchTST), processing time series at their original temporal resolution remains a common and relevant approach in recurrent forecasting models. Several recent works on both full-shot and zero-shot time series forecasting continue to employ point-wise processing without patching, including [Moroshan et al., 2025], which explicitly states that neither patching nor windowing is used, as well as [Graf et al., 2025, Fu et al., 2026, Bhethanabhotla et al., 2024, Behrouz et al., 2024, Alharthi et al., 2024, Kraus et al., 2026]. Thus, we believe that evaluating recurrent architectures at the original temporal resolution remains a relevant setting in its own right.

Second, introducing patching adds another architectural and hyperparameter dimension, particularly the patch length, substantially increasing the experimental search space.


[Moroshan et al., 2025]: V. Moroshan, J. Siems, A., T. Carstensen, F. Hutter, TEMPOPFN: SYNTHETIC PRE-TRAINING OF LINEAR RNNS FOR ZERO-SHOT TIME SERIES FORECASTING

[Graf et al., 2025]: L. Graf, T. Ortner, S. Woźnia, A. Pantazi, FlowState: Sampling-Rate Invariant Time Series Foundation Model with Dynamic Forecasting Horizons

[Fu et al., 2026]: X Fu, Y Li, G Papaioannou, Y Kim, Reverso: Efficient Time Series Foundation Models for Zero-shot Forecasting

[Bhethanabhotla et al., 2024]: S. K. Bhethanabhotla, O. Swelam, J. Siems, D. Salinas, F. Hutter, Mamba4Cast: Efficient Zero-Shot Time Series Forecasting with State Space Models

[Behrouz et al., 2024] A. Behrouz, M. Santacatterina, R. Zabih, Chimera: Effectively modeling multivariate time series with 2-dimensional state space models

[Alharthi et al., 2024] M Alharthi, A Mahmood, xlstmtime: Long-term time series forecasting with xlstm

[Kraus et al., 2026] M. Kraus, F. Divo, D. S. Dhami, K. Kersting, xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories


Weakness 2

On the real-world benchmarks, hyperparameters were tuned on one representative dataset per group and reused across other datasets in that group. Since recurrent architectures can be sensitive to learning rates and state sizes, this reuse might smooth out some genuine differences between models.

We acknowledge that not performing hyperparameter search for each model on each real-world dataset may affect the observed performance differences. We believe that it is a standard practice due to computational resource limitation. However, we would like to emphasize that in our results in the paper for synthetic datasets, we had conducted hyperparameters search on all of the datasets, and obtained conclusions consistent with those drawn from the real-world datasets. We therefore expect dataset-specific tuning to improve absolute performance while having a more limited effect on the relative rankings of the models. To assess this empirically, we conducted additional hyperparameter searches for some models covering different families of expressivity, i.e., LSTM, mLSTM, DeltaProduct, S5, and Mamba-2 on the largest LTSF group, the ETTh1 group for prediction length 96. As expected, hyperparameter tuning improved performance in all cases, but the relative rankings changed only slightly, as shown in the table below

Table (ii)

DatasetModelRank: Transfer from ETTh1Rank: HP SearchTest MSE: Transfer from ETTh1Test MSE: HP Search
etth2DeltaProduct340.50860.3609
etth2LSTM551.67260.3729
etth2Mamba-2420.64920.3370
etth2S5210.47770.3309
etth2mLSTM130.36890.3584
ettm1DeltaProduct550.48140.4419
ettm1LSTM320.42890.4054
ettm1Mamba-2430.45830.4142
ettm1S5110.35880.3121
ettm1mLSTM240.42250.4164
ettm2DeltaProduct350.25530.2061
ettm2LSTM542.53710.2043
ettm2Mamba-2430.25800.1961
ettm2S5210.24520.1781
ettm2mLSTM120.20580.1961
exchangeDeltaProduct220.15640.1456
exchangeLSTM450.18240.2530
exchangeMamba-2330.17740.1622
exchangeS5110.13120.1374
exchangemLSTM540.18400.1795

Weakness 3:

In the regression analysis (Table 2), several architectural features overlap across the 12 models (for example, all nonlinear models in Table 1 are also non-diagonal and state-tracking), which leads to wide confidence intervals and makes it harder to isolate the impact of each individual factor.

The reviewer correctly pointed out a limitation of our study. In fact, we had mentioned it in our Limitations section. We highlighted it in blue in the paper draft, and here we put it to better address the reviewer’s comment:

​​”Third, along the same lines as the previous point, we do not explore all combinations of expressivity axes. This is mainly due to a lack of models which are in some parts of the expressivity space. This impacts our hypothesis testing as it reduces the power at which we can compare some of the expressivity axes.”

Thank you for your response and the additional results. My concerns have been addressed.

Reviewer_djGFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper conducts an empirical evaluation of linear and nonlinear RNN and SSM architectures to assess the impact of recurrence expressivity on time-series forecasting performance. Evaluating these models across controlled synthetic (SynTSBench) and real-world (LTSF, GIFT-Eval) datasets, the authors show that formal recurrence expressivity is not a significant predictor of accuracy. Ultimately, they conclude that forecasting success depends far more on specific dataset properties and architectural inductive biases.

Critical strengths

The paper addresses a compelling research question. The experiments thoroughly evaluate a broad set of architectural approaches on several time series datasets.

Critical weaknesses
  • The implementation is not publicly shared; releasing the source code is essential for a benchmark paper to allow the community to verify and reproduce the findings.
  • While the results correctly highlight dataset properties and inductive biases over expressivity, further empirical analysis clarifying which specific architectural biases and dataset features are most influential would make the study far more complete.
Questions for authors
  • How can one map dataset characteristics to optimal recurrent neural network design choices in time series forecasting?
2 comments

We thank the reviewer for recognizing the importance of our research question and the breadth of our experimental evaluation.

Weakness 1:

The implementation is not publicly shared; releasing the source code is essential for a benchmark paper to allow the community to verify and reproduce the findings.

We agree that making the code publicly available is important for reproducibility. We are currently awaiting authorization to release the code and will make it publicly available as soon as approval is granted, hopefully during the rebuttal period.

Weakness 2:

While the results correctly highlight dataset properties and inductive biases over expressivity, further empirical analysis clarifying which specific architectural biases and dataset features are most influential would make the study far more complete.

We thank the reviewer for raising this important point. It is also similar to another weakness pointed by Reviewer SZAe. We agree that identifying the factors that influence forecasting performance, beyond recurrence expressivity, is an important aspect of understanding the empirical results. To address this, we have added a discussion section (Section 5) to highlight three observations from our experiments and the hypotheses we made based on them.

Here we list our main empirical observations regarding important factors in forecasting that we also included in our answer to Weakness 1 from the Reviewer SZAe:

First, the choice of inductive bias appears to matter for robustness to noise. In particular, on the forecasting tasks with synthetic noisy datasets with periodicity, fixed-transition models such as S5 and LRU, as well as the vanilla RNN (with no input-dependent gates), perform particularly well under high noise. This suggests that simpler, time-invariant state transitions may be advantageous when forecasting signals with strong cyclic structure in noisy observations.

Second, certain architectural design choices may benefit forecasting performance independently of recurrence expressivity. In particular, we observe strong performance from both mLSTM and sLSTM on several real-world datasets. Despite their differences in recurrence expressivity, these models share exponential gating and normalizer states, which we hypothesize may contribute to their strong performance.

Finally, context length emerges as an important factor beyond architectural choices. Our statistical analysis identifies it as a significant factor influencing forecasting performance, suggesting that tuning the context length deserves particular attention during model selection and hyperparameter optimization. (New Table 21 in the paper)

We have incorporated these observations into the new Discussion section, while emphasizing that the proposed architectural explanations are hypotheses that motivate further investigation.

Question:

How can one map dataset characteristics to optimal recurrent neural network design choices in time series forecasting?

Mapping dataset characteristics to the neural network properties is an open question to the best of our knowledge. Nevertheless, architectural components outside the recurrent mechanism can also be designed to exploit specific dataset characteristics. For example, a decomposition layer that separates a time series into trend and seasonal components can provide an appropriate inductive bias when these components are present. This principle is exploited by models such as DLinear, where decomposition is an important part of the architecture and contributes to its strong forecasting performance.

For datasets with temporally stable cyclic patterns, we expect recurrent models with time-invariant transition matrices to capture these patterns more easily than models with input-dependent transitions. This may be due to a more suitable inductive bias, rather than greater expressivity, potentially making the optimization problem easier. Our results on synthetic datasets with noisy periodic signals and multivariate datasets with lagged dependencies between variates provide some support for this hypothesis. However, we acknowledge that this effect is not evident in our real-world experiments, where different time-series characteristics are typically entangled.

Dear Reviewer djGF,

We are also providing the current version of our code in the supplementary material to facilitate access to our implementation and experimental setup. We are continuing to improve its organization and documentation and will provide a cleaned-up version with the final submission.

Thank you for your time and consideration.

Comments on the paper

2 comments

We thank all reviewers for taking the time to review our work and for providing constructive feedback. In this general response, we summarize the additional experiments and revisions we have made to the paper.

Summary of the main revisions:

Additional experiments on:

  • Evaluating the effect of patching
  • Evaluating the effect of hyperparameter transfer

Additional discussion section on:

  • Insights from our empirical study suggesting design choices that matter for time series forecasting

The details of these changes are summarized below:

1. Evaluating the effect of patching. To examine whether our findings depend on using point-wise rather than patch-level sequences, we reran the LTSF experiments using patching following the PatchTST protocol (and by using the hyperparameters from our old non-patched experiment). Although patching generally improves forecasting performance, it does not substantially change the relative performance of the recurrent models on LTSF (New Table 16 in Appendix D), especially leaving mLSTM and sLSTM as best-performing models overall. A summary of changes to the test MSE values (in percentage) for individual models on a subset of datasets is presented in Table (i), in our response to Reviewer 2CjF, Weakness 1. We emphasize that we did all our new experiments with 5 seeds to keep our results statistically reliable.

2. Evaluating the effect of hyperparameter transfer. To examine the impact of transferring hyperparameters from representative to non-representative datasets, we conducted additional hyperparameter optimization experiments on ETTh2, ETTm1, ETTm2, and Exchange, which previously inherited their hyperparameters from ETTh1. Although the absolute performance of several models improved under dataset-specific tuning, their relative rankings changed only slightly. This provides further evidence that our main findings are robust to the hyperparameter transfer used in the original experiments. See Table (ii) in our response to Reviewer 2CjF, Weakness 2. We emphasize that we did these new experiments with 5 seeds to keep our results statistically reliable.

3. Adding a Discussion section on practical insights. In response to the concerns raised by Reviewers djGF02 and SZAe regarding the lack of practical insights beyond our negative results, we added a new Discussion section (Section 5), where besides discussing the important implication of our results for encouraging the use of computationally less expensive diagonal recurrent networks such as Mamba for forecasting tasks, we highlighted two empirical observations and one architectural hypothesis:

  • Context length as a crucial factor: Our statistical analysis identifies context length as an important factor influencing forecasting performance, highlighting the importance of treating it as a tunable hyperparameter. (New Table 21 in the paper)
  • Advantage of Time-invariant transitions: Our synthetic experiments suggest that time-invariant recurrent transitions can be advantageous for forecasting noisy signals with temporally stable cyclic patterns.
  • Advantage of Exponential gating and State Normalization: We hypothesize that these shared design choices may contribute to the strong forecasting performance of the mLSTM and sLSTM architectures on LTSF datasets.
AuthorsOpenReview ↗

Please could the reviewers provide a reply to our paper updates

Dear AC,

As the review period is ending, we would like to ask if the remaining reviewers could provide an reply to our rebuttals, which we worked hard on to have resolved all their concerns.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.