- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents an empirical comparison for time series forecasting across 12 linear and nonlinear architectures, categorized across four axes defined by the authors. The comparison is conducted across both synthetic and real-life benchmarks, and the main takeaways are that more expressivity in an architecture - measured by state tracking capacity - does not present measurable or significant gains on time series forecasting.
Critical strengths
- Overall experimental setup is solid. The equal-shot hyperparameter search, the multi-seed experiments, the synthetic and real-life benchmark evals and the OLS regression analysis show a stable and rigorous evaluation protocol.
- The study covers multiple families of linear and nonlinear RNNs.
Critical weaknesses
-
I can accept that nonlinear recurrences or 'more expressivity' are not required for good forecasting performance, however the paper lacks an explanation on which aspects do matter and influence the forecasting performance. Right now the paper's message is that there is no clear correlation between the examined structural aspects of the architectures and the forecasting performance. An insight as to what does matter would be important for a paper like this.
-
My second point is connected to the first one. I think reporting negative results or "certain architectural choices seem to not correlate with forecasting performance" is a good contribution to the field, but only if it provides insights that are valuable to the community. There are large gaps in performance between models classified in the exact same category according to the 4 criteria devised by the authors. Examples of different average ranks for models that appear in the same model class:
- Cross-Variable synthetic results avg rank: RNN: 1.88, LSTM: 8.5, sLSTM: 9.62, GRU: 6.25.
- LTSF avg. rank: RNN: 5.33, LSTM: 8.78, sLSTM: 3.00, GRU: 4.78.
- Trend synthetic results avg rank: mLSTM: 6.59, Mamba-2: 8.45, Mamba: 2.64.
- LTSF avg. rank: mLSTM: 2.56, Mamba: 8.56, Mamba-2: 8.22.
-
I accept the argument that tasks generally associated with state tracking (parity, permutations etc.) are not relevant to time series forecasting, however: The ranking in the synthetic tasks are at best loosely correlated to real-life benchmark performance. This is also apparent from the above 4 comparisons. This could indicate that the chosen synthetic tasks - while relevant to time series analysis - may not exploit the state tracking capacity the model architectures present.
-
The composite synthetic task, along with Appendix H's table: the authors argue that mLSTM and sLSTM, as well as DeltaNet and Gated DeltaProduct (state tracking counterparts of each other) are 'close' in performance, however it appears to me that both mLSTM and DeltaNet are better than their state tracking counterpart. Moreover, the negative eigenvalues in DeltaNet seem to lower loss only very marginally, but then gated deltaproduct increases it. These are noisy results and are inconclusive to me.
Few more minor points I noticed:
- Griffin does not appear in the table describing the architectures but suddenly appear in the comparison.
- I appreciate the rigorous hyperparam search, however disclosing the final hyperparameters of the models would be important, as it can contain important information about the models. Now the paper's findings are less reproducible.
Questions for authors
- Mamba is best overall on the synthetics (5.00) and near-worst on LTSF (8.56); S5 is best on GIFT and is at best in the middle on LTSF. Which, if any, of the synthetic categories do the authors consider predictive of real-world performance, and on what evidence?
- The current takeaways from the paper is that expressivity does not help. The results on the different benchmarks do not reveal a clear ordering for me, can the authors establish more clean takeaways, with actionable insights on architecture choices for designing more efficient/better performing time series forecasting models?
- mLSTM and Mamba-2 are identical rows in Table 1, yet rank 2.56 vs 8.22 on LTSF. What do the authors attribute this to? can it be due to gating/normalization, the matrix state, tuning budget? If it's something the taxonomy doesn't capture, shouldn't that factor be an axis?
5 comments
Weakness 2:
My second point is connected to the first one. I think reporting negative results or "certain architectural choices seem to not correlate with forecasting performance" is a good contribution to the field, but only if it provides insights that are valuable to the community. There are large gaps in performance between models classified in the exact same category according to the 4 criteria devised by the authors. Examples of different average ranks for models that appear in the same model class:
- Cross-Variable synthetic results avg rank: RNN: 1.88, LSTM: 8.5, sLSTM: 9.62, GRU: 6.25.
- LTSF avg. rank: RNN: 5.33, LSTM: 8.78, sLSTM: 3.00, GRU: 4.78.
- Trend synthetic results avg rank: mLSTM: 6.59, Mamba-2: 8.45, Mamba: 2.64.
- LTSF avg. rank: mLSTM: 2.56, Mamba: 8.56, Mamba-2: 8.22.
We thank the reviewer for emphasizing the importance of providing insights that are valuable to the community. We believe that our findings have practical implications for the design of recurrent forecasting architectures. In particular, they suggest that increasing recurrence expressivity, for example by moving beyond diagonal state transitions, may not be necessary to achieve strong forecasting performance. This motivates the use of simpler and computationally less expensive architectures for time series forecasting. For example, diagonal SSMs such as Mamba are considerably more computationally affordable than models with non-diagonal state transitions, such as DeltaProduct. Since moving beyond diagonal transitions is partly motivated by enabling more expressive state-tracking capabilities, our results suggest that competitive forecasting models, including potentially more efficient time series foundation models, could be developed using diagonal SSMs. We have expanded the Discussion section to elaborate on these implications.
Regarding the performance differences between models within the same expressivity category, we agree that these are important observations. As we also discuss in our response to Weakness 1, architectures within the same category can differ in other design choices that are not captured by our four expressivity criteria. For example, although LSTM and sLSTM belong to the same expressivity category, sLSTM introduces exponential gating and normalizer states, which distinguish its state-update mechanism from that of a standard LSTM without changing its state-tracking expressivity under our criteria. Interestingly, these design choices are shared by mLSTM, which also exhibits strong forecasting performance despite belonging to a different expressivity category. This observation motivates our hypothesis that architectural properties beyond recurrence expressivity, such as exponential gating and state normalization, may play an important role in forecasting performance. We have added this observation to the Discussion section and identified a systematic investigation of these architectural properties as an important direction for future work.
We would also like to emphasize that this difference in models ranking in performance has already been observed for time series forecasting on other types of architectures, for example, [Brigato et al., 2026] also reports different best-performing models on different datasets.
[Brigato et al., 2026]: Lorenzo Brigato, Rafael Morand, Knut Strømmen, Maria Panagiotou, Markus Schmidt, Stavroula Mougiakakou, There are no Champions in Supervised Long-Term Time Series Forecasting
Weakness 3:
I accept the argument that tasks generally associated with state tracking (parity, permutations etc.) are not relevant to time series forecasting, however: The ranking in the synthetic tasks are at best loosely correlated to real-life benchmark performance. This is also apparent from the above 4 comparisons. This could indicate that the chosen synthetic tasks - while relevant to time series analysis - may not exploit the state tracking capacity the model architectures present.
We agree that model rankings vary between synthetic and real-world datasets (again also in accordance with earlier observations from [Brigato et al., 2026]), and even across the two real-world benchmarks. However, an important observation is that our main finding remains consistent: models with simpler recurrence mechanisms and lower expressivity remain among the strongest performers across these settings. We would also like to clarify that, when comparing synthetic and real-world results, the overall ranking across synthetic datasets may provide a more representative picture than the ranking on any individual synthetic dataset, such as the Cross-Variable dataset highlighted in Weakness 2.
Nevertheless, we find the reviewer's point important. The fact that performance rankings do not consistently transfer from synthetic to real-world datasets raises the question of whether existing synthetic forecasting tasks adequately capture the properties of real-world time series, and, in particular, whether they sufficiently exploit the state-tracking capabilities of recurrent architectures. We see this as an underexplored question in the time series forecasting literature. Indeed, whether state-tracking capabilities are necessary for forecasting, and whether the synthetic tasks commonly used in time series research require such capabilities, were among the questions motivating our study. Although tasks such as parity and permutation tracking are generally considered canonical state-tracking problems, their relevance to forecasting is unclear. Our experiments suggest that the state-tracking capabilities studied in the literature may not translate into consistent forecasting benefits on the synthetic and real-world datasets we considered. This is particularly relevant given that several recent recurrent forecasting studies have motivated more expressive architectures partly through their state-tracking capabilities.
We agree that the synthetic tasks considered in our study may not fully exploit these capabilities. However, this possibility does not undermine our findings: rather, it highlights that the forecasting tasks currently used in empirical evaluations, including commonly used synthetic datasets, may not require the additional state-tracking expressivity offered by more sophisticated recurrent architectures. Investigating more structured synthetic processes, for example those generated by chaotic dynamical systems, and examining whether their forecasting benefits from more expressive state-tracking mechanisms, is an interesting direction for future work.
Weakness 4:
The composite synthetic task, along with Appendix H's table: the authors argue that mLSTM and sLSTM, as well as DeltaNet and Gated DeltaProduct (state tracking counterparts of each other) are 'close' in performance, however it appears to me that both mLSTM and DeltaNet are better than their state tracking counterpart. Moreover, the negative eigenvalues in DeltaNet seem to lower loss only very marginally, but then gated deltaproduct increases it. These are noisy results and are inconclusive to me.
We thank the reviewer for pointing this out. We agree that the results do not consistently support the claim that these model pairs perform comparably. For the composite synthetic task (Table 3), looking at test MSE rather than average rankings provides a more complete picture, as the performance advantage is relatively evenly distributed between mLSTM and sLSTM, and similarly between DeltaNet and DeltaProduct. We therefore consider the ranking metric complementary to the test MSE when interpreting these results. However, we acknowledge that the results in Appendix G (the old Appendix H) show a clearer preference for the less expressive counterparts, i.e., DeltaNet and negative-Deltanet over DeltaProduct. We will revise our description to reflect this asymmetry rather than characterize their performance as comparable.
We would also note that the stronger forecasting performance of DeltaNet compared to DeltaProduct is consistent with observations in recent literature, including [Fu et al. (2026)]. This provides additional context for our observation that the more expressive DeltaProduct architecture does not necessarily offer a forecasting advantage over DeltaNet.
[Fu et al., 2026]: X Fu, Y Li, G Papaioannou, Y Kim, Reverso: Efficient Time Series Foundation Models for Zero-shot Forecasting
We thank the reviewer for recognizing the rigor of our experimental evaluation, including the equal-budget hyperparameter search, multi-seed experiments, diverse synthetic and real-world benchmarks, and statistical analysis. We also appreciate the reviewer’s acknowledgment of the breadth of recurrent architectures covered in our study.
Weakness 1:
We thank the reviewer for raising this important point. This point has also been addressed in our response to Weakness 2 from Reviewer djGF. We agree that identifying the factors that influence forecasting performance, beyond recurrence expressivity, is an important aspect of understanding the empirical results. To address this, we have added a Discussion section (Section 5) to highlight three observations from our experiments and the hypotheses we made based on them.
First, the choice of inductive bias appears to matter for robustness to noise. In particular, on the forecasting tasks with synthetic noisy datasets with periodicity, fixed-transition models such as S5 and LRU, as well as the vanilla RNN, perform particularly well under high noise. This suggests that simpler, time-invariant state transitions may be advantageous when forecasting signals with strong cyclic structure in noisy observations.
Second, certain architectural design choices may benefit forecasting performance independently of recurrence expressivity. In particular, we observe strong performance from both mLSTM and sLSTM on several real-world datasets. Despite their differences in recurrence expressivity, these models share exponential gating and normalizer states, which we hypothesize may contribute to their strong performance.
Finally, context length emerges as an important factor beyond architectural choices. Our statistical analysis identifies it as a significant factor influencing forecasting performance, suggesting that tuning the context length deserves particular attention during model selection and hyperparameter optimization. (New Table 21 in the paper)
We have incorporated these observations into the new Discussion section, while emphasizing that the proposed architectural explanations are hypotheses that motivate further investigation.