- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work investigates whether the recent superiority of Transformers, linear models, and Mamba over RNNs in time-series forecasting stems from exposing richer historical representations to the decoder rather than the sequence modeling mechanism itself. Unlike conventional recurrent baselines that compress all history into a single terminal state, modern architectures preserve multi-step historical features, granting greater expressive power. The authors validate this hypothesis by feeding the full sequence of recurrent hidden states to the prediction head, which substantially boosts RNN performance. Although this final-state bottleneck has long been explored in fields like NLP, systematically isolating it within modern time-series pipelines remains meaningful.
Critical strengths
-
This paper investigates whether the performance gap between traditional RNNs and modern architectures like Transformers stems from a previously overlooked limitation: the restricted scope of historical information accessible to the prediction head, where relying on a single hidden state for decoding creates a severe information bottleneck. This represents a very interesting insight.
-
To address the question at hand, the authors conducted a simple yet effective experiment: they retained the hidden states during the recurrence and fed them into the final forecasting head, while controlling the number of hidden states accessible to that head, in order to investigate how the visibility of historical representations affects the model's predictive performance.
Critical weaknesses
-
Limited generalizability across diverse Transformer architectures: While the paper's core insight is compelling, the experimental validation requires further expansion to support its broad claims. Because the authors aim to demonstrate that full representation accessibility enables RNNs to match or outperform Transformer-based methods in general, evaluating this encoder-replacement strategy across a wider range of Transformer architectures is necessary. Currently, the controlled framework relies exclusively on the PatchTST pipeline (simply replacing the PatchTST backbone with an RNN). It remains unclear whether this conclusion holds when substituting the self-attention layers in other representative Transformer paradigms (e.g., iTransformer or Crossformer). Relying on a single architectural template makes it difficult to establish that the findings are truly generalizable across the broader Transformer family.
-
Outdated Transformer baselines: The baseline methods evaluated in the experiments need to be significantly updated and expanded. Given that a primary goal of this paper is to benchmark recurrent models against Transformer-based forecasters, the included Transformer baselines are limited to architectures from 2023 and earlier. Notably, although the authors cite more recent advances such as iTransformer (2024) in the related work section, they omit them from the empirical comparisons. Including state-of-the-art Transformer baselines from 2024–2026 is essential to convincingly establish the competitiveness of the proposed FSR framework.
-
Limitation of the Channel-Independent assumption: From a mechanistic standpoint, restricting the investigation solely to the CI setting is overly limiting. Transformers do more than just enrich the input to the prediction head; they also capture inter-channel dependencies, a core focus in multivariate time-series forecasting. The authors should further explore and discuss whether multi-state readout remains effective when multi-channel information is jointly modeled.
Questions for authors
See the weakness above
2 comments
Response to Reviewer m4Nf (Part2)
Table2
| Models | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Electricity | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | |
| Ours | 0.412 | 0.419 | 0.355 | 0.390 | 0.221 | 0.294 | 0.176 | 0.289 | ||
| TimeBridge | 0.373 | 0.405 | 0.330 | 0.371 | 0.156 | 0.251 | ||||
| TimeXer | 0.429 | 0.432 | 0.370 | 0.405 | 0.377 | 0.392 | 0.173 | 0.257 | 0.176 | 0.289 |
| iTransformer | 0.445 | 0.449 | 0.348 | 0.384 | 0.243 | 0.312 |
Q3:Does multi-state readout remain effective when inter-channel information is jointly modeled, rather than only under channel independence?
A3: Thank you for this suggestion. Multi-state readout remains effective under joint channel modeling. We construct Channel-Dependent (CD) variants of our model and SegRNN, and compare Last-State and Full-State Readout for each model. While keeping the respective encoders and other architectural components unchanged, we evaluate the impact of different readout ranges on forecasting performance.
Table3
| Models | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Electricity | Traffic | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | |
| Ours-CD-Last | 0.447 | 0.451 | 0.378 | 0.409 | 0.370 | 0.405 | 0.240 | 0.312 | 0.216 | 0.325 | 0.627 | 0.432 |
| Ours-CD-Full | 0.411 | 0.421 | 0.356 | 0.386 | 0.340 | 0.379 | 0.223 | 0.295 | 0.172 | 0.284 | 0.447 | 0.336 |
| SegRNN-CD-Last | 0.420 | 0.433 | 0.351 | 0.392 | 0.346 | 0.387 | 0.235 | 0.312 | 0.212 | 0.305 | 0.793 | 0.412 |
| SegRNN-CD-Full | 0.415 | 0.429 | 0.349 | 0.393 | 0.332 | 0.373 | 0.222 | 0.297 | 0.190 | 0.290 | 0.701 | 0.393 |
As shown, Ours-CD consistently achieves better MSE and MAE with Full-State Readout across all six datasets, while SegRNN-CD exhibits the same trend in the vast majority of evaluated settings. Overall, these results suggest that the benefits of multi-state readout are not limited to channel-independent settings and can also extend to joint modeling of inter-channel dependencies, although the extent of the improvements may vary across architectures and datasets. These additional results have been included in the appendix of the revised manuscript.
We sincerely appreciate this valuable suggestion. The corresponding experimental results and discussion have been added to
Response to Reviewer m4Nf (Part1)
We heartily thank you for this encouraging and constructive feedback. Through your valuable suggestions, we have conducted additional experiments and expanded the corresponding discussions. Your comments have helped us further extend and enrich the content of our manuscript.
Q1: Does the encoder-replacement finding extend beyond the PatchTST pipeline, for example to Crossformer or iTransformer?
A1: Yes, we use iTransformer to demonstrate the generalizability of our model across different Transformer architectures.
To further address your concern, we conduct self-attention replacement experiments within the iTransformer framework. Specifically, we replace the self-attention modules in each encoder block with bidirectional GRUs operating along the variate-token dimension, while retaining the original variate-token embedding, residual connections, normalization, FFN, and prediction head. We adopt bidirectional GRUs because iTransformer represents each variate as a token, and bidirectional modeling allows each variate representation to incorporate contextual information from both directions, rather than being restricted to a single direction. The results show that the GRU-based variants achieve competitive performance with the original iTransformer, providing additional evidence for the feasibility of recurrent module replacement beyond the PatchTST pipeline.
Table1
As shown, Full outperforms Last in the vast majority of evaluated settings, providing additional evidence for the feasibility of recurrent module replacement across architectural templates beyond PatchTST. The corresponding experimental results and discussions have been added to the appendix. We thank the reviewer for this valuable suggestion, which has helped us further extend our experimental evaluation.
We sincerely appreciate this valuable suggestion. Following your recommendation, we have added the corresponding experimental results and discussion to <u>Table 15 in Appendix A.9 (the revised manuscript)</u>. This suggestion has been instrumental in broadening our experimental evaluation and improving the overall quality of the manuscript.
Q2: The benchmark focuses on older Transformer architectures. Please include recent models from 2024–2026, particularly iTransformer.
A2: Thank you for your suggestion. To further evaluate the competitiveness of our model against more recent Transformer models, we have included three additional baselines: TimeBridge (2025), TimeXer (2024), and iTransformer (2024). The experiments are conducted with a prediction horizon of 192, and the results are presented in the table below.
The proposed model outperforms all three baselines on both ETTh1 and ETTh2, and outperforms TimeXer and iTransformer on ETTm1. Specifically, compared with iTransformer, our model achieves lower MSE and MAE on four out of the five datasets. Compared with TimeXer, it achieves better performance on three datasets. These results demonstrate that our model remains competitive against more recent Transformer-based forecasting models.