acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Recurrent Models for Long-Term Time Series Forecasting through Representation Accessibility

Abstract

Transformer-based architectures have become prominent in long-term time series forecasting (LTSF), and their empirical advantages are often attributed to self-attention. This paper revisits that attribution by identifying a less visible architectural confounder in common LTSF comparisons: representation accessibility. In many direct multi-horizon recurrent baselines, the prediction module is conditioned on a compressed recurrent summary, typically the final encoder state, or on a decoder initialized from that state. In contrast, patch-based Transformer forecasters such as PatchTST expose representations from multiple temporal positions directly to a non-autoregressive prediction head. This difference changes not only the sequence modeling mechanism, but also which historical representations are accessible at prediction time. To isolate this factor, we introduce a Full-State Readout (FSR) framework built from standard recurrent encoders. FSR keeps key ingredients of strong LTSF forecasters fixed, including patching, channel independence, normalization, and a Flatten–Linear prediction head, while varying only the number of recurrent hidden states exposed to the head. Across six LTSF benchmarks, FSR with a GRU backbone matches or outperforms representative Transformer baselines. Expanding representation accessibility generally reduces forecasting errors for GRU, and full-state readout also improves vanilla RNN and LSTM encoders. These results show that the recurrent–Transformer gap in LTSF cannot be attributed solely to recurrence versus self-attention. More broadly, they suggest that final-state bottlenecks can systematically underestimate recurrent models, and that representation accessibility should be controlled when comparing sequence modeling architectures for forecasting.

Primary area:
learning on time series and dynamical systems
Keywords:
Long-Term Time Series Forecasting, Recurrent Neural Networks, Representation Accessibility, Full-State Readout, Transformer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_m4NfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates whether the recent superiority of Transformers, linear models, and Mamba over RNNs in time-series forecasting stems from exposing richer historical representations to the decoder rather than the sequence modeling mechanism itself. Unlike conventional recurrent baselines that compress all history into a single terminal state, modern architectures preserve multi-step historical features, granting greater expressive power. The authors validate this hypothesis by feeding the full sequence of recurrent hidden states to the prediction head, which substantially boosts RNN performance. Although this final-state bottleneck has long been explored in fields like NLP, systematically isolating it within modern time-series pipelines remains meaningful.

Critical strengths
  1. This paper investigates whether the performance gap between traditional RNNs and modern architectures like Transformers stems from a previously overlooked limitation: the restricted scope of historical information accessible to the prediction head, where relying on a single hidden state for decoding creates a severe information bottleneck. This represents a very interesting insight.

  2. To address the question at hand, the authors conducted a simple yet effective experiment: they retained the hidden states during the recurrence and fed them into the final forecasting head, while controlling the number of hidden states accessible to that head, in order to investigate how the visibility of historical representations affects the model's predictive performance.

Critical weaknesses
  1. Limited generalizability across diverse Transformer architectures: While the paper's core insight is compelling, the experimental validation requires further expansion to support its broad claims. Because the authors aim to demonstrate that full representation accessibility enables RNNs to match or outperform Transformer-based methods in general, evaluating this encoder-replacement strategy across a wider range of Transformer architectures is necessary. Currently, the controlled framework relies exclusively on the PatchTST pipeline (simply replacing the PatchTST backbone with an RNN). It remains unclear whether this conclusion holds when substituting the self-attention layers in other representative Transformer paradigms (e.g., iTransformer or Crossformer). Relying on a single architectural template makes it difficult to establish that the findings are truly generalizable across the broader Transformer family.

  2. Outdated Transformer baselines: The baseline methods evaluated in the experiments need to be significantly updated and expanded. Given that a primary goal of this paper is to benchmark recurrent models against Transformer-based forecasters, the included Transformer baselines are limited to architectures from 2023 and earlier. Notably, although the authors cite more recent advances such as iTransformer (2024) in the related work section, they omit them from the empirical comparisons. Including state-of-the-art Transformer baselines from 2024–2026 is essential to convincingly establish the competitiveness of the proposed FSR framework.

  3. Limitation of the Channel-Independent assumption: From a mechanistic standpoint, restricting the investigation solely to the CI setting is overly limiting. Transformers do more than just enrich the input to the prediction head; they also capture inter-channel dependencies, a core focus in multivariate time-series forecasting. The authors should further explore and discuss whether multi-state readout remains effective when multi-channel information is jointly modeled.

Questions for authors

See the weakness above

2 comments
AuthorsOpenReview ↗

Response to Reviewer m4Nf (Part1)

We heartily thank you for this encouraging and constructive feedback. Through your valuable suggestions, we have conducted additional experiments and expanded the corresponding discussions. Your comments have helped us further extend and enrich the content of our manuscript.

Q1: Does the encoder-replacement finding extend beyond the PatchTST pipeline, for example to Crossformer or iTransformer?

A1: Yes, we use iTransformer to demonstrate the generalizability of our model across different Transformer architectures.

To further address your concern, we conduct self-attention replacement experiments within the iTransformer framework. Specifically, we replace the self-attention modules in each encoder block with bidirectional GRUs operating along the variate-token dimension, while retaining the original variate-token embedding, residual connections, normalization, FFN, and prediction head. We adopt bidirectional GRUs because iTransformer represents each variate as a token, and bidirectional modeling allows each variate representation to incorporate contextual information from both directions, rather than being restricted to a single direction. The results show that the GRU-based variants achieve competitive performance with the original iTransformer, providing additional evidence for the feasibility of recurrent module replacement beyond the PatchTST pipeline.

Table1

Modelspred_lenETTh1 ETTh2 ETTm1 ETTm2 Electricity Traffic 
  MSEMAEMSEMAEMSEMAEMSEMAEMSEMAEMSEMAE
iTransformer_GRU_Last960.3950.4090.3090.3570.3400.3760.1780.2640.1460.2430.3780.268
 1920.4530.4420.3860.4040.3800.3960.2490.3090.1650.2620.3910.276
 3360.5240.4820.4260.4370.4170.4180.3210.3520.1930.2880.4100.284
 7200.5530.5170.4290.4490.4790.4530.4100.4060.2290.3180.4430.296
iTransformer_GRU_Full960.3860.4040.2970.3490.3340.3690.1770.2640.1410.2380.3780.264
 1920.4400.4340.3780.3980.3750.3900.2470.3090.1620.2570.3910.272
 3360.5120.4760.4170.4310.4080.4110.3130.3500.1840.2770.4150.282
 7200.5510.5130.4240.4450.4750.4500.4110.4060.2130.3030.4450.295

As shown, Full outperforms Last in the vast majority of evaluated settings, providing additional evidence for the feasibility of recurrent module replacement across architectural templates beyond PatchTST. The corresponding experimental results and discussions have been added to the appendix. We thank the reviewer for this valuable suggestion, which has helped us further extend our experimental evaluation.

We sincerely appreciate this valuable suggestion. Following your recommendation, we have added the corresponding experimental results and discussion to <u>Table 15 in Appendix A.9 (the revised manuscript)</u>. This suggestion has been instrumental in broadening our experimental evaluation and improving the overall quality of the manuscript.

Q2: The benchmark focuses on older Transformer architectures. Please include recent models from 2024–2026, particularly iTransformer.

A2: Thank you for your suggestion. To further evaluate the competitiveness of our model against more recent Transformer models, we have included three additional baselines: TimeBridge (2025), TimeXer (2024), and iTransformer (2024). The experiments are conducted with a prediction horizon of 192, and the results are presented in the table below.

The proposed model outperforms all three baselines on both ETTh1 and ETTh2, and outperforms TimeXer and iTransformer on ETTm1. Specifically, compared with iTransformer, our model achieves lower MSE and MAE on four out of the five datasets. Compared with TimeXer, it achieves better performance on three datasets. These results demonstrate that our model remains competitive against more recent Transformer-based forecasting models.

AuthorsOpenReview ↗

Response to Reviewer m4Nf (Part2)

Table2

ModelsETTh1 ETTh2 ETTm1 ETTm2 Electricity 
 MSEMAEMSEMAEMSEMAEMSEMAEMSEMAE
Ours0.4120.4190.3550.3900.335‾\underline{0.335}0.375‾\underline{0.375}0.2210.2940.1760.289
TimeBridge0.417‾\underline{0.417}0.422‾\underline{0.422}0.3730.4050.3300.3710.177‾\underline{0.177}0.264‾\underline{0.264}0.1560.251
TimeXer0.4290.4320.3700.4050.3770.3920.1730.2570.1760.289
iTransformer0.4450.4490.365‾\underline{0.365}0.399‾\underline{0.399}0.3480.3840.2430.3120.167‾\underline{0.167}0.266‾\underline{0.266}

Q3:Does multi-state readout remain effective when inter-channel information is jointly modeled, rather than only under channel independence?

A3: Thank you for this suggestion. Multi-state readout remains effective under joint channel modeling. We construct Channel-Dependent (CD) variants of our model and SegRNN, and compare Last-State and Full-State Readout for each model. While keeping the respective encoders and other architectural components unchanged, we evaluate the impact of different readout ranges on forecasting performance.

Table3

ModelsETTh1 ETTh2 ETTm1 ETTm2 Electricity Traffic 
 MSEMAEMSEMAEMSEMAEMSEMAEMSEMAEMSEMAE
Ours-CD-Last0.4470.4510.3780.4090.3700.4050.2400.3120.2160.3250.6270.432
Ours-CD-Full0.4110.4210.3560.3860.3400.3790.2230.2950.1720.2840.4470.336
SegRNN-CD-Last0.4200.4330.3510.3920.3460.3870.2350.3120.2120.3050.7930.412
SegRNN-CD-Full0.4150.4290.3490.3930.3320.3730.2220.2970.1900.2900.7010.393

As shown, Ours-CD consistently achieves better MSE and MAE with Full-State Readout across all six datasets, while SegRNN-CD exhibits the same trend in the vast majority of evaluated settings. Overall, these results suggest that the benefits of multi-state readout are not limited to channel-independent settings and can also extend to joint modeling of inter-channel dependencies, although the extent of the improvements may vary across architectures and datasets. These additional results have been included in the appendix of the revised manuscript.

We sincerely appreciate this valuable suggestion. The corresponding experimental results and discussion have been added to Table 16 in Appendix A.10 of the revised manuscript.‾\underline{\text{Table 16 in Appendix A.10 of the revised manuscript.}}

Reviewer_KATMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper investigates representation accessibility as a confounding factor in comparisons between recurrent models and transformers for long-term forecasting. It proposes FSR, exposing all recurrent hidden states to a non autoregressive prediction head. Experiments on six datasets show that GRU, LSTM, and vanilla RNN models benefit from broader state access and can achieve performance comparable to PatchTST.

Critical strengths

paper addresses an important issue in architectural comparisons: separating the quality of learned representations from their accessibility to the prediction head. The experiments across three recurrent architectures provide convincing evidence that final state readout can unnecessarily restrict forecasting performance. The results also challenge simplistic claims about the superiority of self attention.

Critical weaknesses

The main concern is that increasing the number of accessible states also increases the number of prediction head parameters substantially. In Eq 7, the head dimension grows linearly with the number of accessible states. Therefore, the improvements in Tab. 3 cannot be attributed exclusively to representation accessibility without parameter matched controls. The technical novelty is limited, since accessing multiple recurrent hidden states and concatenating them for prediction are established ideas. Also, the reported experiments provide no clear seed level uncertainty estimates, and the statistical analysis does not establish a significant advantage over PatchTST. The work is a useful empirical observation, but its central causal explanation remains insufficiently isolated.

Questions for authors

Can the authors provide a parameter-matched comparison between final-state and full state readout to separate representation accessibility from increased head capacity? Were the results reproduced across independent training seeds, and how stable are the reported improvements?

6 comments
AuthorsOpenReview ↗

Response to Reviewer KATM (Part1)

Many thanks to you for providing thorough insightful comments and recognizing the importance of distinguishing representation quality from the representations available to the forecasting head.

Q1: Can the authors provide a parameter-matched comparison between final-state and full-state readout? The current K-sweep does not isolate the central causal explanation.

A1: Thank you for the suggestion. The original K-sweep simultaneously changes the number of accessible states and the prediction-head parameter count. To disentangle their effects, we introduce an approximately parameter-matched final-state readout baseline, FSR-GRU-MatchedMLP. It uses the same patch embedding and GRU encoder as the original FSR-GRU, but feeds only the final hidden state into a two-layer MLP whose parameter count approximately matches that of the K=42 Flatten–Linear prediction head. All experiments use the same data splits and training configurations, with three random seeds. We report the mean and standard deviation of MSE and MAE.

Table1

ModelSettingMetricETTh1 ETTh2 ETTm1 ETTm2 Electricity Traffic 
   MeanStdMeanStdMeanStdMeanStdMeanStdMeanStd
OursFinal (K=1)MSE0.4490.0030.3680.0020.3660.0010.234‾\underline{0.234}0.0020.2180.0010.6230.002
  MAE0.4500.0010.4040.0010.4030.0000.306‾\underline{0.306}0.0020.3270.0010.4280.001
 Full (K=42)MSE0.4130.0010.3530.0010.3340.0010.2230.0010.1760.0000.4370.001
  MAE0.4210.0010.3890.0010.3750.0010.2960.0010.2890.0000.3230.002
FSR-GRU-MatchedMLPK=1MSE0.415‾\underline{0.415}0.0020.361‾\underline{0.361}0.0010.353‾\underline{0.353}0.0010.2350.0040.199‾\underline{0.199}0.0000.497‾\underline{0.497}0.003
  MAE0.428‾\underline{0.428}0.0010.397‾\underline{0.397}0.0010.392‾\underline{0.392}0.0010.306‾\underline{0.306}0.0030.310‾\underline{0.310}0.0000.366‾\underline{0.366}0.001

As shown, FSR-GRU-MatchedMLP improves upon the original K=1 configuration in 10 out of 12 dataset–metric pairs, indicating that increasing prediction-head capacity can partially improve forecasting performance. However, it consistently underperforms our model with K=42 across all six datasets in terms of both MSE and MAE. These results suggest that the performance gains from expanding the accessible state range cannot be explained solely by the increased prediction-head parameter count; effectively exploiting representations from multiple temporal positions also contributes to the improvements.

The corresponding experiment and its analysis have been added to Appendix A.7 of the revised manuscript.‾\underline{\text{added to Appendix A.7 of the revised manuscript.}}

Q2: Were the results reproduced across independent training seeds, and how stable are the improvements?

A2: Yes, the results reproduced across independent training seeds.To evaluate the reproducibility and stability of our findings, we conducted the following additional experiments using three independent random seeds. For Table 2 (as presented in the original paper), we conducted three-seed experiments on the ETTh1 dataset at a prediction horizon of 96, comparing SegRNN, PatchTST, GRU, LSTM, and our model. For Table 3 (as presented in the original paper), we repeated the K-sweep experiments to examine the performance of FSR-GRU with different numbers of accessible hidden states, namely K = 1, 14, 28. For Table 4 (as presented in the original paper), we repeated the experiments using three independent random seeds and reported the corresponding results. A detailed analysis is presented below:

Table2

ModelsETTh1 seed1ETTh1 seed1  ETTh1 seed2ETTh1 seed2ETTh1 seed3ETTh1 seed3Avg MSEStd MSEAvg MAEStd MAE
 MSEMAEMSEMAEMSEMAEMSEMAEMSEMAE
Ours0.3780.3980.379‾\underline{0.379}0.3990.3780.3980.3780.0010.3980.001
GRU0.4460.4560.4500.4580.4500.4580.4490.0020.4570.001
LSTM0.4500.4600.4510.4600.4500.4590.4500.0010.4600.001
SegRNN0.3910.4160.3890.4150.3900.4150.3900.0010.4150.001
PatchTST0.380‾\underline{0.380}0.405‾\underline{0.405}0.3780.403‾\underline{0.403}0.379‾\underline{0.379}0.405‾\underline{0.405}0.379‾\underline{0.379}0.0010.404‾\underline{0.404}0.001
AuthorsOpenReview ↗

Response to Reviewer KATM (Part2)

Table2 (continued)

ModelsETTh2 seed1ETTh2 seed1ETTh2 seed2ETTh2 seed2ETTh2 seed3ETTh2 seed3Avg MSEStd MSEAvg MAEStd MAE
 MSEMAEMSEMAEMSEMAEMSEMAEMSEMAE
Ours0.290‾\underline{0.290}0.3460.2850.3420.289‾\underline{0.289}0.3460.288‾\underline{0.288}0.0030.3450.002
GRU0.3890.4270.3900.4260.3870.4190.3890.0020.4240.004
LSTM0.4260.4470.4250.4460.4100.4390.4200.0090.4440.004
SegRNN0.2850.347‾\underline{0.347}0.2850.346‾\underline{0.346}0.2880.350‾\underline{0.350}0.2860.0020.348‾\underline{0.348}0.002
PatchTST0.3020.3620.315‾\underline{0.315}0.3670.3190.3700.3120.0090.3660.004

As can be seen in Table2, at a prediction horizon of 96, our model achieves the best average MSE and MAE on ETTh1 (0.378 ± 0.001 and 0.398 ± 0.001), slightly outperforming PatchTST. On ETTh2, our model (0.288 ± 0.003 and 0.345 ± 0.002) outperforms PatchTST, GRU, and LSTM on both metrics, while achieving lower MAE but slightly higher MSE than SegRNN. The small standard deviations further indicate stable performance across three random seeds.

Table3

ModelSettingMetricETTh1 ETTh2 ETTm1 ETTm2 Electricity Traffic 
   MeanStdMeanStdMeanStdMeanStdMeanStdMeanStd
OursFinalMSE0.4490.0030.3680.0020.3660.0010.2340.0020.2180.0010.6230.002
  MAE0.4500.0020.4040.0010.4030.0000.3060.0020.3270.0010.4280.001
Ours14MSE0.4250.0000.3560.0010.3410.0010.2300.0010.1880.0010.4810.002
  MAE0.4260.0000.3900.0000.3830.0010.3020.0010.3010.0010.3500.001
Ours28MSE0.420‾\underline{0.420}0.0010.354‾\underline{0.354}0.0020.340‾\underline{0.340}0.0050.224‾\underline{0.224}0.0010.180‾\underline{0.180}0.0000.452‾\underline{0.452}0.000
  MAE0.423‾\underline{0.423}0.0010.3880.0020.378‾\underline{0.378}0.0010.2960.0020.293‾\underline{0.293}0.0000.330‾\underline{0.330}0.000
Ours42MSE0.4130.0010.3530.0010.3340.0010.2230.0010.1760.0000.4370.001
  MAE0.4210.0010.389‾\underline{0.389}0.0010.3740.0010.2960.0010.2890.0000.3230.002

As in Table 3, we repeated the K-sweep experiments to examine how the forecasting performance of FSR-GRU varies with the number of accessible hidden states (K). The results averaged over three random seeds remain consistent with the main trends reported in the original submission. As the number of accessible hidden states (K) increases, the proposed model generally achieves lower forecasting errors. In particular, compared with Final-State Readout (K=1), Full-State Readout (K=42) improves both MSE and MAE across all six datasets (12/12 comparisons). Although the results are not strictly monotonic for some intermediate K values, the overall improvement trend remains consistent.

AuthorsOpenReview ↗

Response to Reviewer KATM (Part3)

Table4

ModelSettingMetricETTh1 ETTh2 ETTm1 ETTm2 Electricity Traffic 
   MeanStdMeanStdMeanStdMeanStdMeanStdMeanStd
OursFinalMSE0.4490.0030.3680.0020.3660.0010.2340.0020.2180.0010.6230.002
  MAE0.4500.0010.4040.0010.4030.0000.3060.0020.3270.0010.4280.001
 FullMSE0.413‾\underline{0.413}0.0010.353‾\underline{0.353}0.0010.3340.0010.2230.0010.176‾\underline{0.176}0.0000.437‾\underline{0.437}0.001
  MAE0.421‾\underline{0.421}0.0010.389‾\underline{0.389}0.0010.375‾\underline{0.375}0.0010.296‾\underline{0.296}0.0010.2890.0000.323‾\underline{0.323}0.002
FSR-LSTMFinalMSE0.4590.0060.3710.0020.3690.0020.2460.0040.2220.0000.6350.003
  MAE0.4570.0020.4050.0010.4040.0010.3120.0010.3310.0000.4340.001
 FullMSE0.4180.0010.3610.0080.335‾\underline{0.335}0.0010.227‾\underline{0.227}0.0000.1740.0000.4340.000
  MAE0.4230.0020.3910.0040.3740.0010.296‾\underline{0.296}0.0010.2860.0000.3190.000
FSR-RNNFinalMSE0.4300.0000.3570.0010.3520.0010.2310.0010.2190.0000.6430.002
  MAE0.4380.0000.3900.0000.3900.0010.3050.0020.3270.0000.4290.001
 FullMSE0.4090.0000.3470.0070.3360.0000.2230.0010.176‾\underline{0.176}0.0020.4390.000
  MAE0.4170.0000.3850.0050.3740.0010.2940.0020.288‾\underline{0.288}0.0020.3250.001

As in Table 4, Full-State Readout consistently outperforms Final-State Readout across all three recurrent encoders (RNN, GRU, and LSTM), achieving lower forecasting errors in all 36 dataset–metric comparisons. The standard deviations are relatively small in most experimental settings, and the improvements remain consistent across the evaluated configurations, providing further evidence of the stability of our findings across different random seeds.

Q3: Accessing and concatenating recurrent hidden states are established ideas, so the technical novelty is limited.

A3: We agree that accessing and exploiting multiple recurrent hidden states is not new in itself. Previous studies, such as TPA-LSTM, WITRAN, P-sLSTM, and RWKV-TS, have utilized multi-state information through attention-based aggregation, multi-directional state fusion, or full-state flattening.

However, our contribution does not lie in introducing a new multi-state readout operation or proposing another SOTA architecture. Rather, our central objective is to investigate whether the performance disadvantage traditionally associated with recurrent models in long-term forecasting is partly attributable to the restricted state accessibility of conventional final-state readout, rather than to recurrent modeling itself. To this end, we treat the accessible range of hidden states (K) as an explicit research variable and systematically examine its effect on forecasting performance within an aligned LTSF pipeline. Specifically, by retaining PatchTST's patching, channel independence, and Flatten–Linear prediction head while replacing only the Transformer encoder with a GRU, we examine whether broader access to recurrent representations can narrow the performance gap between recurrent and Transformer-based models.

The K-sweep results in Table 3 (as presented in the original paper) and the consistent findings across RNN, GRU, and LSTM backbones in Table 4 (as presented in the original paper) support the importance of representation accessibility across different recurrent architectures. Together with the encoder replacement experiments, these results show that broader access to recurrent hidden states can improve forecasting performance and enable recurrent models to achieve competitive results with Transformers under the evaluated settings. We therefore position representation accessibility as an important factor in understanding and evaluating recurrent forecasting architectures, rather than claiming novelty in the multi-state readout operation itself.

Reviewer_pZPMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that comparisons between recurrent and Transformer forecasters are confounded by representation accessibility, i.e. how many encoder states the prediction head can read: recurrent baselines typically read the final hidden state, whereas PatchTST flattens all patch tokens into a linear head. The proposed Full-State Readout (FSR) is a PatchTST-style pipeline (instance normalisation, channel independence, overlapping patches with P=16P=16, S=8S=8, positional encoding, Flatten–Linear head; Sec. 3) with the Transformer encoder replaced by a recurrent encoder (RNN, GRU or LSTM, dh=128d_h=128) over the N=42N=42 patches of a 336-step input, where the head receives the last KK hidden states. Validation consists of a benchmark of FSR-GRU against 11 baselines under one shared training recipe (Tables 2, 7), where it is best or tied-best on 11 of 12 averaged cells and has the best Friedman rank but is not separable from PatchTST or SegRNN (Fig. 3); a sweep of K∈{1,14,28,42}K\in\{1,14,28,42\} at T=192T=192 for FSR-GRU and PatchTST (Table 3); a final-versus-full readout comparison for three recurrent cells (Table 4); appendix studies on segmentations, SegRNN variants, ESN/CNN encoders, patch overlap and alternative readouts (Tables 8–12); and an efficiency comparison (Table 5).

Critical strengths

Strength 1. The KK-sweep is applied symmetrically to both encoder families under an aligned pipeline (Sec. 4.3, Tables 3, 4, 8). Input, patching, head form and training recipe are held fixed while the number of exposed states varies (together with head size; see Weakness 2). The effect is large and consistent: on Traffic at T=192T=192 the GRU goes from 0.625 to 0.437 MSE, LSTM from 0.638 to 0.434 and vanilla RNN from 0.643 to 0.438; full readout beats final readout in all 36 cells of Table 4, at both N=28N=28 and N=84N=84 (Table 8) and in SegRNN's own encoder at N=42N=42 (Table 9); and the recurrent-versus-Transformer verdict flips with the readout configuration alone (4 of 12 wins at K=1K=1, 8 wins and 3 ties at K=42K=42). This is the right kind of experiment for the question posed.

Strength 2. The confounder framing is a legitimate contribution to benchmark practice. Where recurrent baselines are run with final-state readout and no patching, the paper is right that such numbers say little about recurrence per se, and Table 1 states the distinction compactly.

Critical weaknesses

Weakness 1. The headline comparison rests on baselines trained far below their published strength, and FSR-GRU itself does not beat the published numbers (Tables 2, 3, 7; Sec. 4.1).

The paper's PatchTST uses the published PatchTST/42 lookback and patching (L=336L=336, P=16P=16, S=8S=8) and, on Electricity, Traffic and ETTm, the published architecture, yet its averaged MSE is 0.448 vs 0.396 on Traffic, 0.192 vs 0.162 on Electricity and 0.455 vs 0.417 on ETTh1 (four-horizon averages of Nie et al., 2023, Table 3); at Traffic T=192T=192 it reports 0.437 where the published value is 0.385, so the sentence in Sec. 1 that FSR-GRU is "matching the reported MSE of full-state PatchTST" refers to the authors' own run. The SegRNN re-run is further off: averaged Traffic MSE/MAE 0.865/0.424 vs 0.596/0.255 in Lin et al. (five-run average, L=720L=720), Electricity 0.227 vs 0.161. A plausible contributor is the shared recipe of Sec. 4.1: learning rate 10−410^{-4} halved after every epoch (about 10−710^{-7} after ten epochs; the Time-Series-Library default, but used by neither PatchTST's nor SegRNN's published recipe), batch 16 and one model size for all datasets, whereas the official PatchTST scripts use a one-cycle schedule on ETTm/Electricity/Traffic, batch 24/32 on Traffic/Electricity and a much smaller model (d=16d=16, 4 heads) on ETTh1/h2, and SegRNN uses d=512d=512 with per-dataset learning rates up to 3×10−33\times10^{-3}.

FSR-GRU itself (Electricity 0.191, Traffic 0.448) does not beat the published PatchTST/42 average on any of the 12 dataset–metric pairs and sits behind published DLinear at L=336L=336 (0.166 and 0.434), which the paper cites but omits from Table 2. The paper reports neither the number of runs nor any variance, while several counted wins are 0.001–0.003 apart (Table 3, K=42K=42: Electricity 0.176 vs 0.178; Table 2: Electricity 0.191 vs 0.192, ETTm1 0.359 vs 0.361) and three K=42K=42 cells are exact ties. Because every number in Tables 2, 3 and 7 depends on this recipe, the comparative conclusions cannot be assessed from the current draft.

Weakness 2. The factor the paper names is not isolated from head capacity, the paper's own appendix points to a different explanation, and the pipeline is not new.

Sec. 4.3 states that the forecasting head is "fixed" while KK varies, but a Flatten–Linear head (Eqs. 6–7) has T⋅K⋅dhT\cdot K\cdot d_h weights: about 24.6K at K=1K=1 and 1.03M at K=42K=42 for T=192T=192, which is about 91% of the 1.134M parameters in Table 5 if that configuration matches the forecasting one. The KK-sweep therefore varies accessibility and model size together. The alternative readouts in Appendix A.6 do not resolve this: single-query cross-attention and mean pooling feed one dhd_h-vector to the head and are capacity-matched to K=1K=1, and on ETTh1, Electricity and Traffic they are much worse than plain final-state readout (ETTh1 MSE 0.677 and 0.694 vs 0.452), which is hard to explain if the query can learn to attend to the last state and calls for a convergence check. Two further results motivate a no-recurrence control: under full readout vanilla RNN, GRU and LSTM are within 0.014 MSE of each other on every dataset (Table 4), and an echo state network with a fixed reservoir matches or beats the trained GRU on Electricity and Traffic, though not on ETT (Table 10). With an identity encoder the pipeline reduces to a linear map of the normalised input, i.e. an RLinear-type model, and that control is absent.

Finally, the pipeline (patching, channel independence, recurrent encoder over patches, flatten-all-states linear head, L=336L=336) is, apart from the recurrent block, non-overlapping patches and the absence of instance normalisation, the architecture of P-sLSTM (Kong et al., AAAI 2025), which is not cited and reports better Electricity numbers (0.130/0.148/0.165/0.199 vs 0.160/0.176/0.193/0.233 in Table 7); RWKV-TS (Hou and Yu, 2024, arXiv) and patch-based Mamba models with a flatten head (e.g. Bi-Mamba+) follow the same recipe. None of these varies KK, so the sweep itself is not anticipated, but Table 1, which presents FSR as the only recurrent method with patching, multi-state readout and a non-autoregressive head, omits them. The Related Work also stops at SegRNN: WITRAN (Jia et al., NeurIPS 2023) already passes multiple recurrent states (all column-direction states plus the last row-direction state) to fully connected layers rather than decoding from one state, and PRformer (Yu et al., Neural Networks 2025) is a segment-level GRU with final-state readout, an instance of the interface the paper criticises.

Questions for authors
  1. [clarification] Why are the re-run PatchTST and SegRNN results so far from their published values? Please report Tables 2 and 3 with each baseline trained under its official recipe and state whether the KK-sweep conclusion survives.

  2. [clarification] How many runs underlie each number? Please report mean and standard deviation over at least three seeds for Tables 2–4 and recompute the win tallies with variance taken into account.

  3. [suggestion] Please add (a) a capacity-matched final-state readout (K=1K=1 with an MLP head matched to the full-state head's parameter count) and (b) a no-recurrence encoder (identity or per-patch MLP) with the same head, together with DLinear/RLinear under your protocol. If (b) matches FSR-GRU, the claims about what recurrence adds would need to be narrowed.

  4. [clarification] On ETTh1, Electricity and Traffic, why are mean pooling and single-query cross-attention over all 42 states (Table 12) so much worse than reading only the last state (Table 4)? Were they trained identically, and did they converge?

  5. [clarification] Please state the input length and number of recurrent layers in Sec. 4.1, describe the LSTM/GRU/TPA-LSTM baselines of Table 2 (and their 8.1M parameters in Table 5), explain how N=28N=28 and N=84N=84 in Table 8 are obtained, and discuss the relation to P-sLSTM, RWKV-TS, WITRAN and the multi-state readouts already reported for MQ-RNN and TPA-LSTM.

6 comments
AuthorsOpenReview ↗

Response to Reviewer pZPM (Part1)

We sincerely thank you for your thorough and rigorous review. In response to your valuable suggestions, we have conducted additional experiments and further refined our discussions.

Q1: Why are the re-run PatchTST and SegRNN results substantially weaker than their published results? Please use official training recipes for Tables 2 and 3 and verify whether the K-sweep conclusion survives.

A1: Thank you for the careful comparison and constructive comments. We acknowledge the discrepancies between our reproduced results and those reported in the original papers, and agree that further clarification of the experimental protocols and the scope of our conclusions is necessary.

First, we would like to clarify the central objective of this work. Our goal is not to achieve new SOTA results, but to investigate an often-overlooked factor in long-term time series forecasting: Representation Accessibility. Specifically, we examine whether the performance gap of conventional recurrent models is partly attributable to the practice of using only the final hidden state for prediction, rather than to the recurrent mechanism itself. By retaining PatchTST's patching, channel independence, and Flatten–Linear prediction head while replacing only the Transformer encoder with a GRU, we find that making recurrent hidden states fully accessible enables the GRU to achieve forecasting performance comparable to Transformers under the same experimental settings. Together with the K-sweep results in Table 3 (as presented in the original paper) and the consistent findings across RNN, GRU, and LSTM in Table 4 (as presented in the original paper), these results support our central argument regarding representation accessibility, independently of whether the models achieve the best absolute performance.

Regarding the discrepancies in baseline reproduction, we conduct our experiments using the commonly used Time-Series-Library (https://github.com/thuml/Time-Series-Library) , a benchmark framework for time series forecasting, and adopt a unified training protocol to minimize the influence of experimental differences on our mechanism analysis, rather than conducting model-specific hyperparameter searches to optimize each baseline individually. We acknowledge that this setting may prevent some models from achieving their best officially reported performance. In particular, the official PatchTST implementation uses drop_last=True, whereas Time-Series-Library defaults to False; this difference in evaluation protocols may also affect the results. Furthermore, SegRNN incorporates a specialized parallel multi-step forecasting design, which is not entirely equivalent to the Encoder-Only + Flatten–Linear architecture that we focus on. Differences between its official configuration and our unified training settings may therefore explain part of the performance discrepancy. We thus consider these results suggestive of potential sensitivity to training strategies and implementation details, but insufficient to establish conclusions about model robustness.

Q2: How many runs underlie each number? Please report mean and standard deviation over at least three seeds for Tables 2–4 and recompute the win tallies with variance taken into account.

A2: The results in the original submission were based on a single run using the fixed random seed configured in the Time-Series-Library (TSL), with the same seed applied consistently across all models.

To further address your concern regarding reproducibility and stability across different random seeds, we conducted additional experiments using three independent random seeds. Due to time constraints, we performed partial three-seed evaluations for Tables 2 and 3 (as presented in the original paper), while all experiments in Table 4 (as presented in the original paper) were fully repeated across three random seeds. Specifically, for Table 2 (as presented in the original paper), we conducted three-seed experiments on the ETTh1 dataset at a prediction horizon of 96, comparing SegRNN, PatchTST, GRU, LSTM, and our model. For Table 3 (as presented in the original paper), we repeated the K-sweep experiments to examine the performance of FSR-GRU with different numbers of accessible hidden states, namely K = 1, 14, 28, and 42. For Table 4 (as presented in the original paper), all experiments were fully repeated using three independent random seeds. The results obtained so far remain consistent with the main findings of the original submission. We report the corresponding means and standard deviations below, followed by a detailed analysis.

AuthorsOpenReview ↗

Response to Reviewer pZPM (Part2)

Table1

ModelsETTh1 seed1ETTh1 seed1  ETTh1 seed2ETTh1 seed2ETTh1 seed3ETTh1 seed3Avg MSEStd MSEAvg MAEStd MAE
 MSEMAEMSEMAEMSEMAEMSEMAEMSEMAE
Ours0.3780.3980.379‾\underline{0.379}0.3990.3780.3980.3780.0010.3980.001
GRU0.4460.4560.4500.4580.4500.4580.4490.0020.4570.001
LSTM0.4500.4600.4510.4600.4500.4590.4500.0010.4600.001
SegRNN0.3910.4160.3890.4150.3900.4150.3900.0010.4150.001
PatchTST0.380‾\underline{0.380}0.405‾\underline{0.405}0.3780.403‾\underline{0.403}0.379‾\underline{0.379}0.405‾\underline{0.405}0.379‾\underline{0.379}0.0010.404‾\underline{0.404}0.001
ModelsETTh2 seed1ETTh2 seed1ETTh2 seed2ETTh2 seed2ETTh2 seed3ETTh2 seed3Avg MSEStd MSEAvg MAEStd MAE
 MSEMAEMSEMAEMSEMAEMSEMAEMSEMAE
Ours0.290‾\underline{0.290}0.3460.2850.3420.289‾\underline{0.289}0.3460.288‾\underline{0.288}0.0030.3450.002
GRU0.3890.4270.3900.4260.3870.4190.3890.0020.4240.004
LSTM0.4260.4470.4250.4460.4100.4390.4200.0090.4440.004
SegRNN0.2850.347‾\underline{0.347}0.2850.346‾\underline{0.346}0.2880.350‾\underline{0.350}0.2860.0020.348‾\underline{0.348}0.002
PatchTST0.3020.3620.315‾\underline{0.315}0.3670.3190.3700.3120.0090.3660.004

As in Table 1, at a prediction horizon of 96, our model achieves the best average MSE and MAE on ETTh1 (0.378 ± 0.001 and 0.398 ± 0.001), slightly outperforming PatchTST. On ETTh2, our model (0.288 ± 0.003 and 0.345 ± 0.002) outperforms PatchTST, GRU, and LSTM on both metrics, while achieving lower MAE but slightly higher MSE than SegRNN. The small standard deviations further indicate stable performance across three random seeds.

Table2

ModelSettingMetricETTh1 ETTh2 ETTm1 ETTm2 Electricity Traffic 
   MeanStdMeanStdMeanStdMeanStdMeanStdMeanStd
OursFinalMSE0.4490.0030.3680.0020.3660.0010.2340.0020.2180.0010.6230.002
  MAE0.4500.0020.4040.0010.4030.0000.3060.0020.3270.0010.4280.001
Ours14MSE0.4250.0000.3560.0010.3410.0010.2300.0010.1880.0010.4810.002
  MAE0.4260.0000.3900.0000.3830.0010.3020.0110.3010.0010.3500.001
Ours28MSE0.420‾\underline{0.420}0.0010.354‾\underline{0.354}0.0020.340‾\underline{0.340}0.0050.224‾\underline{0.224}0.0010.180‾\underline{0.180}0.0000.452‾\underline{0.452}0.000
  MAE0.423‾\underline{0.423}0.0010.3880.0020.378‾\underline{0.378}0.0010.2960.0020.293‾\underline{0.293}0.0000.330‾\underline{0.330}0.000
Ours42MSE0.4130.0010.3530.0010.3340.0010.2230.0010.1760.0000.4370.001
  MAE0.4210.0010.389‾\underline{0.389}0.0010.3740.0010.2960.0010.2890.0000.3230.002

As in Table 2, the three-seed results remain consistent with the main findings of the original submission. As the number of accessible states (K) increases, the proposed model generally achieves lower forecasting errors. Compared with final-state readout (K=1), full-state readout (K=42) improves both MSE and MAE across all six datasets . For example, on Traffic, MSE decreases from 0.623 ± 0.002 to 0.437 ± 0.001. The standard deviations are small across the evaluated settings (at most 0.005), indicating limited variability across the three runs. Overall, these results support the stability of the main conclusion that broader representation accessibility improves recurrent forecasting performance under the evaluated settings.

AuthorsOpenReview ↗

Response to Reviewer pZPM (Part3)

Table3

ModelSettingMetricETTh1 ETTh2 ETTm1 ETTm2 Electricity Traffic 
   MeanStdMeanStdMeanStdMeanStdMeanStdMeanStd
OursFinalMSE0.4490.0030.3680.0020.3660.0010.2340.0020.2180.0010.6230.002
  MAE0.4500.0010.4040.0010.4030.0000.3060.0020.3270.0010.4280.001
 FullMSE0.413‾\underline{0.413}0.0010.353‾\underline{0.353}0.0010.3340.0010.2230.0010.176‾\underline{0.176}0.0000.437‾\underline{0.437}0.001
  MAE0.421‾\underline{0.421}0.0010.389‾\underline{0.389}0.0010.375‾\underline{0.375}0.0010.296‾\underline{0.296}0.0010.2890.0000.323‾\underline{0.323}0.002
FSR-LSTMFinalMSE0.4590.0060.3710.0020.3690.0020.2460.0040.2220.0000.6350.003
  MAE0.4570.0020.4050.0010.4040.0010.3120.0010.3310.0000.4340.001
 FullMSE0.4180.0010.3610.0080.335‾\underline{0.335}0.0010.227‾\underline{0.227}0.0000.1740.0000.4340.000
  MAE0.4230.0020.3910.0040.3740.0010.296‾\underline{0.296}0.0010.2860.0000.3190.000
FSR-RNNFinalMSE0.4300.0000.3570.0010.3520.0010.2310.0010.2190.0000.6430.002
  MAE0.4380.0000.3900.0000.3900.0010.3050.0020.3270.0000.4290.001
 FullMSE0.4090.0000.3470.0070.3360.0000.2230.0010.176‾\underline{0.176}0.0020.4390.000
  MAE0.4170.0000.3850.0050.3740.0010.2940.0020.288‾\underline{0.288}0.0020.3250.001

As in Table 3, the results averaged over three random seeds are consistent with those reported in the original submission, with only minor numerical differences. The standard deviations are also small across most settings, indicating that the observed improvements are stable across runs. Moreover, the advantage of Full-State readout over Final-State readout remains consistent across RNN, GRU, and LSTM backbones, supporting the robustness of our main finding.

Q3: Please add (a) a capacity-matched final-state readout (K = 1 with an MLP head matched to the fullstate head's parameter count) and (b) a no-recurrence encoder (identity or per-patch MLP) with the same head, together with DLinear/RLinear under your protocol. ****If (b) matches FSR-GRU, the claims about what recurrence adds would need to be narrowed.

A3: Thank you for this important suggestion. As suggested, we have included and compared these models, leading to the following conclusions. We report MSE and MAE as the mean ± standard deviation over three random seeds.

(1) Separating readout range from head capacity. FSR-GRU-MatchedMLP reads only the final GRU state but uses an MLP head with approximately the same parameter count as the K=42 head. It improves upon the original K=1 model on 10 of 12 dataset–metric pairs, showing that head capacity matters. Nevertheless, its mean MSE and MAE remain worse than those of FSR-GRU-K42 on all six datasets. These results suggest that the performance gains from expanding the accessible state range cannot be explained solely by the increased prediction-head parameter count; effectively exploiting representations from multiple temporal positions also contributes to the improvements.

Table4

ModelSettingMetricETTh1 ETTh2 ETTm1 ETTm2 Electricity Traffic 
   MeanStdMeanStdMeanStdMeanStdMeanStdMeanStd
OursFinal (K=1)MSE0.4490.0030.3680.0020.3660.0010.234‾\underline{0.234}0.0020.2180.0010.6230.002
  MAE0.4500.0010.4040.0010.4030.0000.306‾\underline{0.306}0.0020.3270.0010.4280.001
 Full (K=42)MSE0.4130.0010.3530.0010.3340.0010.2230.0010.1760.0000.4370.001
  MAE0.4210.0010.3890.0010.3750.0010.2960.0010.2890.0000.3230.002
FSR-GRU-MatchedMLPK=1MSE0.415‾\underline{0.415}0.0020.361‾\underline{0.361}0.0010.353‾\underline{0.353}0.0010.2350.0040.199‾\underline{0.199}0.0000.497‾\underline{0.497}0.003
  MAE0.428‾\underline{0.428}0.0010.397‾\underline{0.397}0.0010.392‾\underline{0.392}0.0010.306‾\underline{0.306}0.0030.310‾\underline{0.310}0.0000.366‾\underline{0.366}0.001

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Our work asks whether conventional recurrent models underperform in long-term forecasting because of recurrence itself or limited historical representations accessible to the prediction head. To investigate this question, we systematically examine Representation Accessibility, focusing on how the number of recurrent hidden states available to the prediction head affects forecasting performance. Our key insight is that broader access to historical representations can substantially improve forecasting performance, making accessibility an important factor in recurrent–Transformer comparisons.

We sincerely thank all reviewers for their insightful reviews and encouraging assessments. Specifically, the reviewers commented:

  • The paper offers “a very interesting insight” (Reviewer m4Nf) and addresses “an important issue in architectural comparisons” (Reviewer KATM). Its framing of representation accessibility as a confounding factor is recognized as “a legitimate contribution to benchmark practice” (Reviewer pZPM).
  • The controlled K-sweep is recognized as “the right kind of experiment for the question posed” (Reviewer pZPM). The experiments across three recurrent architectures provide “convincing evidence” (Reviewer KATM), and the approach is described as “simple yet effective” (Reviewer m4Nf).
  • The paper is “clear and easy to follow” and “interesting or useful” (Reviewers pZPM, KATM, and m4Nf).

We also thank the reviewers for their constructive concerns, which motivated the following revisions:

1. Additional Ablation Studies on Representation Accessibility and Model Capacity [Reviewers: pZPM, KATM]

We added a parameter-matched final-state readout baseline (FSR-GRU-MatchedMLP) (suggested by pZPM, KATM) to distinguish representation accessibility from increased prediction-head capacity. The two models differ by just 144 parameters (approximately 0.013%). Although increasing head capacity improves performance, Full-State Readout (K=42) achieves lower MSE and MAE across all six datasets.

We also introduced two non-recurrent controls (FSR-Identity and FSR-PatchMLP) (suggested by pZPM), together with DLinear and RLinear, to investigate the role of recurrent encoding. These experiments clarify the roles of accessibility, head capacity, and encoding mechanisms.

2. Extensions to Different Architectures and Channel-Dependent Settings [Reviewer: m4Nf]

We extended our investigation beyond PatchTST by replacing iTransformer's self-attention modules with bidirectional GRUs. The resulting models are competitive, and Full-State Readout generally outperforms Last-State Readout across evaluated settings.

We also conducted Channel-Dependent (CD) experiments with our model and SegRNN. Full-State Readout improves both metrics across all six datasets for our CD variant, with similar improvements observed for SegRNN-CD in most settings. These results extend the evaluation beyond channel-independent modeling.

3. Comparisons with Recent Transformer Baselines [Reviewer: m4Nf]

We included TimeBridge (2025), TimeXer (2024), and iTransformer (2024) at a prediction horizon of 192. Our model achieves lower MSE and MAE than all three on ETTh1 and ETTh2 and remains competitive on other evaluated datasets, providing a more up-to-date benchmark comparison.

4. Experimental Robustness, Readout Analysis, and Further Clarifications [Reviewers: pZPM, KATM]

To address reproducibility concerns, we conducted experiments using three independent random seeds, reporting means and standard deviations for selected baseline comparisons, K-sweeps, and recurrent backbones. Full-State Readout outperforms Final-State Readout across RNN, GRU, and LSTM in all 36 dataset–metric comparisons (suggested by pZPM, KATM), supporting the stability of our findings.

We also repeated the Last-State, Mean Pooling, and single-query Cross-Attention experiments (suggested by pZPM) under identical training protocols and examined the training and validation loss curves to assess stable training behavior. These results clarify how different readout mechanisms organize and preserve temporal information.

Furthermore, we clarified the unified training protocol, baseline configurations, parameter counts, and patch-number calculations, and expanded related-work discussions to include P-sLSTM, RWKV-TS, Bi-Mamba+, WITRAN, and PRformer (suggested by pZPM). We also refined our contribution: we systematically investigate representation accessibility rather than claim a novel multi-state readout operation or statistically superior SOTA performance.

Our extensive efforts to address all feedback and refine the manuscript resulted in a slight delay in our revision submission. We sincerely appreciate the reviewers' understanding and patience, and hope the revised manuscript addresses their concerns and contributes to the research community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.