acceptodds
Under review as a conference paper at ICLR 2027

TimeRelay: Relation-Aware Recurrent Variable Modeling for Multivariate Time Series Forecasting

Abstract

Multivariate time series forecasting requires effective modeling of temporal dynamics within individual variables and predictive dependencies across variables. A representative class of cross-variable modeling approaches explicitly constructs variable-to-variable interactions, resulting in a candidate interaction space that grows quadratically with the number of variables. At the same time, the predictive relevance of different variable pairs can be highly non-uniform. Motivated by these observations, we propose TimeRelay, a relation-aware recurrent framework that models cross-variable dependencies from the perspective of predictive information propagation rather than explicitly materializing dense pairwise interactions. TimeRelay first constructs variable-wise temporal representations from historical observations and then applies Relation-Aware State Propagation (RASP). For each variable, RASP constructs a compact multivariate context and uses the variable–context relation to adaptively control recurrent state updates. RASP therefore propagates predictive information across variables through evolving recurrent states without explicitly constructing a dense variable-to-variable interaction matrix. Its affine recurrent form enables work-efficient parallel evaluation with linear arithmetic work in the number of variables and logarithmic dependency depth. Extensive experiments on diverse multivariate benchmarks demonstrate the effectiveness, efficiency, and scalability of TimeRelay.

Primary area:
learning on time series and dynamical systems
Keywords:
Relation-Aware State Propagation, Multivariate Time Series Forecasting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m8RBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TimeRelay combines multiscale temporal embeddings with a leave-one-out context that controls bidirectional propagation across variables. Its gates and candidate updates are independent of the previous recurrent state, allowing exact evaluation through an associative affine scan. The paper evaluates forecasting accuracy, alternative mixers, computational scaling and traversal choices on ten datasets.

Critical strengths
  1. The recurrence and its parallel evaluation fit together cleanly. Associative composition changes the execution schedule rather than approximating the model. Table 9 compares implementations of the same recurrence and reports a PEMS07 epoch-time reduction from 350 to 33 seconds.

  2. The evaluation contains useful controlled comparisons, not just complete-model rankings. Tables 3–4 separate temporal encoding, context and alternative mixers; the direction and plug-in experiments provide further support. The reported gains are strongest on traffic datasets and Solar, rather than uniformly across all long-horizon benchmarks.

Critical weaknesses
  1. The evidence does not fully separate global contextualization from recurrent propagation. PlainScan tests recurrence without relation-aware context, and Context-BiGRU tests a different recurrence with context. Neither is a context-conditioned, nonrecurrent block with the same encoder, relation features and output budget. That control would directly test what propagation adds after the other variables have already been summarized. Table 4 also ties RASP with Mamba and Context-BiGRU on PEMS07, while Table 6 sometimes favors mean context. These are useful results, but Section 5.4's exclusion of context or recurrence alone as explanations is stronger than the controls establish. Robustness to separately trained traversal orders does not resolve this attribution question.

  2. The accuracy and efficiency evidence needs a clearer connection to the evaluated configurations. The D=64 comparison is a legitimate equal-width control, but main forecasting runs use D=512; Tables 8–9 therefore do not establish the main model's accuracy–cost trade-off. The attention backend and measurement protocol should be reported, with a fused SDPA/FlashAttention comparison where supported [1]. The paper also omits per-horizon results and variability for close comparisons, and Figure 8 does not specify its forecast horizon or aggregation. Please provide the corresponding run/configuration records, including baseline provenance and the patience-3 versus released patience-5 setting, rather than leaving these details implicit.

References

[1] Dao et al. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. 2022. https://arxiv.org/abs/2205.14135

Questions for authors
  1. What happens when the leave-one-out context and relation-conditioned features are retained but recurrent propagation is replaced by a parameter-matched nonrecurrent update?

  2. Can you provide horizon-level results, the settings behind Figure 8, and representative D=512 cost measurements? Please identify the attention backend, actual training configurations and which baseline entries were rerun.

5 comments
AuthorsOpenReview ↗

Response to Reviewer m8RB (Part I)

Thank you for your constructive comments. We have addressed all your comments below with additional experiments.


W1 & Q1. Comparison of RASP with global-context mixing without recurrence.

RASP uses compact multivariate context to condition state transitions, while recurrent states carry predictive information accumulated through individual variable updates. The relation-aware gate determines how much propagated information is retained and how much of the current variable's candidate is incorporated.

To directly isolate the contribution of propagation, we evaluated a parameter-matched, context-conditioned nonrecurrent variant. We retained the same leave-one-out context encoder, relation features, gates, candidate projections, output projection, and residual connection, together with the same temporal embedding and prediction head. Only recurrent state propagation was removed.

Specifically, RASP computes the forward update as hi→=gi→⊙hi−1→+(1−gi→)⊙ui→h_i^{\rightarrow} = g_i^{\rightarrow} \odot h_{i-1}^{\rightarrow} + (1-g_i^{\rightarrow}) \odot u_i^{\rightarrow}, whereas the nonrecurrent variant computes hi→=(1−gi→)⊙ui→h_i^{\rightarrow} = (1-g_i^{\rightarrow}) \odot u_i^{\rightarrow}. The backward branch is modified in the same way. Thus, each variable still receives a relation-conditioned update, but no state is propagated between variables.

Results. As shown in Table 13 below, RASP achieves lower MSE on all ten datasets and lower MAE on nine, with a tie on PEMS07. For example, it reduces MSE from 0.215 to 0.192 on Solar, from 0.439 to 0.427 on Traffic, and from 0.723 to 0.716 on METR-LA. Because the nonrecurrent variant retains the same context encoder and relation features, these results support the additional predictive benefit of recurrent state propagation beyond context-conditioned local updates within the shared forecasting backbone.

Table 13. Comparison of RASP with global-context mixing without recurrence.

MethodMetricPEMS03PEMS04PEMS07PEMS08PEMSD7ECLTrafficSolarPEMS-BAYMETR-LA
TimeRelay (RASP)MSE0.0900.0880.0730.1140.3240.1670.4270.1920.3620.716
MAE0.1970.1950.1680.2010.3130.2610.2810.2490.2770.474
+ Only global-contextMSE0.0960.0910.0740.1160.3260.1690.4390.2150.3700.723
MAE0.2030.2000.1680.2030.3150.2650.2860.2640.2800.483

W2.1 & Q2.1. Horizon-level results

We report TimeRelay's MSE and MAE at each prediction horizon in Table 14 below, together with the unweighted average over the four horizons for each dataset.

Table 14. Forecasting results of TimeRelay at each prediction horizon.

MethodHorizonMetricPEMS03PEMS04PEMS07PEMS08PEMSD7PEMS-BAYMETR-LAHorizonECLTrafficSolar
TimeRelay12MSE0.0590.0690.0520.0730.2350.2480.432960.1370.3910.173
MAE0.1610.1730.1450.1700.2570.2270.3310.2330.2650.227
24MSE0.0730.0790.0610.0950.3070.3270.6101920.1580.4160.188
MAE0.1780.1840.1570.1900.3010.2610.4120.2520.2750.244
48MSE0.1000.0930.0800.1200.3660.4050.8123360.1740.4320.199
MAE0.2090.2030.1780.2090.3350.2960.5220.2700.2830.262
96MSE0.1290.1090.0980.1670.3890.4671.0087200.1980.4670.208
MAE0.2390.2210.1920.2340.3590.3250.6310.2900.3020.262
AverageMSE0.0900.0880.0730.1140.3240.3620.716Average0.1670.4270.192
MAE0.1970.1950.1680.2010.3130.2770.4740.2610.2810.249

W2.2 & Q2.2. Experimental settings for Fig. 8

In Fig. 8, we vary the input length L∈{96,192,336,576,720}L \in \{96, 192, 336, 576, 720\} while fixing the forecast horizon at 720 to evaluate forecasting performance over a longer prediction window. The reported results are for this single horizon, not averaged across multiple forecast horizons. Across all input lengths, we use a batch size of 32, a maximum of 15 training epochs, an early-stopping patience of 5, and a learning rate of 0.001.

AuthorsOpenReview ↗

Response to Reviewer m8RB (Part II)


W2.3 & Q2.3. Attention backend, measurement protocol, and comparison with fused attention.

To connect the efficiency comparison to the main forecasting results in Tables 1–2 of the original paper, we evaluated Attention, FlashAttention, and RASP at D = 512 on ECL, Traffic, and PEMS07, using four model layers for all three methods. We also evaluated D = 64, 128, and 256 while keeping the layer count fixed at four to examine how the accuracy–cost trade-off changes with embedding width.

Attention and FlashAttention backend. The Attention baseline uses an explicit implementation that computes QKTQK^T, applies softmax, and multiplies the resulting attention weights by VV. The FlashAttention baseline uses PyTorch's scaled_dot_product_attention with the FLASH_ATTENTION backend explicitly enabled. Its normalization, residual connections, and dropout settings are aligned with the Attention baseline. The CUDA FlashAttention backend used in our implementation requires FP16 or BF16 Q/K/V tensors. We therefore used CUDA automatic mixed precision (AMP) with FP16 autocasting for all three methods to ensure a consistent precision policy. All experiments were conducted on an NVIDIA H20 GPU with a batch size of 32. We retained the same temporal embedding and prediction head and replaced only the cross-variable mixer. We report the number of trainable parameters, training time per epoch, and GPU memory usage.

Time cost. Table 15 below shows that RASP achieves the lowest or tied-lowest epoch time across all tested datasets and embedding dimensions, although the runtime improvements are modest. At D = 512, the epoch times for Attention, FlashAttention, and RASP are 32/32/31 s on ECL, 30/29/28 s on Traffic, and 38/33/32 s on PEMS07. Attention benefits from highly parallel GPU matrix operations, while FlashAttention further reduces memory traffic through fused kernels. These advantages may keep attention fast at the tested variable counts despite its dense pairwise computation. Since epoch time includes the shared temporal embedding and prediction head, mixer-level runtime differences may be less pronounced.

Memory cost. FlashAttention uses less GPU memory than RASP, while RASP uses substantially less memory than explicit Attention. At D = 64, FlashAttention and RASP have similar memory usage across the three datasets despite their different information-exchange mechanisms. FlashAttention achieves this through tiled, fused computation and recomputation during backpropagation, avoiding storage of the full C×CC \times C attention matrices. Its attention intermediates require linear storage in C at a fixed batch size and embedding dimension D, although it retains dense pairwise computation. RASP's intermediate storage also scales linearly with C under these fixed conditions, but its memory gap relative to FlashAttention widens as D increases. RASP maintains additional D-dimensional representations for context construction, relation-aware gates, candidates, and bidirectional propagation, which may contribute to its higher memory usage at larger D. In future work, we will explore optimizations along both the variable and embedding dimensions to reduce the dependence of memory usage on D and further lower RASP's linear memory cost in C.

Performance. RASP achieves the lowest or tied-lowest MSE and the lowest MAE across all tested settings. The gains vary across datasets and dimensions: at D = 64, RASP reduces MSE from 0.096 to 0.085 on PEMS07 and from 0.490 to 0.478 on Traffic compared with both attention baselines. At D = 512, it reduces MSE from 0.431 to 0.429 on Traffic and from 0.076 to 0.075 on PEMS07. On ECL, MSE is tied at 0.167, while MAE decreases from 0.263 to 0.262. These results characterize RASP's accuracy–cost trade-off on the evaluated datasets: competitive training time, substantially lower GPU memory usage than explicit Attention, memory usage close to FlashAttention at D = 64, and improved forecasting accuracy.

AuthorsOpenReview ↗

Response to Reviewer m8RB (Part III)


Table 15. Comparison of Attention, FlashAttention, and RASP across embedding dimensions.

DDatasetMetricAttentionFlashAttentionRASPDatasetAttentionFlashAttentionRASPDatasetAttentionFlashAttentionRASP
512ECLParameters (M)5.1205.1209.333Traffic5.4485.4489.661PEMS075.4615.4619.674
Time (s)323231302928383332
Memory (MB)28151887234710739409547971116941594893
MSE0.1670.1670.1670.4310.4310.4290.0760.0760.075
MAE0.2630.2630.2620.2870.2870.2820.1720.1720.170
256ECLParameters (M)1.4141.4142.472Traffic1.6041.6042.662PEMS071.6121.6122.669
Time (s)313030292827353131
Memory (MB)2471154517671028734253845992733773797
MSE0.1680.1680.1670.4430.4430.4380.0780.0780.076
MAE0.2630.2630.2620.2970.2970.2900.1790.1790.172
128ECLParameters (M)0.4440.4440.711Traffic0.5640.5640.831PEMS070.5690.5690.835
Time (s)313030282827353130
Memory (MB)2357152116239733317333271012731933349
MSE0.1750.1750.1730.4640.4640.4530.0840.0840.081
MAE0.2680.2680.2670.3130.3130.3050.1870.1870.181
64ECLParameters (M)0.1910.1910.259Traffic0.2860.2860.354PEMS070.2900.2900.358
Time (s)303029282826333130
Memory (MB)2409146114779625294530311001929773055
MSE0.1830.1830.1820.4900.4900.4780.0960.0960.085
MAE0.2750.2750.2730.3330.3330.3270.2020.2020.188

W2.4 & Q2.4. Training configurations, baseline provenance, and rerun settings.

The training configurations of TimeRelay are described in Appendix B.2, Implementation Details. The released code uses an early-stopping patience of 5. For the baseline experiments we reran, we used a maximum of 15 training epochs, an early-stopping patience of 5, learning rate of 0.001, dropout of 0.1, hidden dimension of 512, and batch size of 32, unless otherwise specified below. We distinguish results taken from the original papers from those obtained through our reruns as follows.

  • iTransformer, SOFTS, and TQNet: We used the results reported in their original papers.

  • CycleNet: We reran the CycleNet/MLP variant using the dataset-specific scripts under scripts/CycleNet in the official repository. We retained the default model-specific settings and applied the common training settings above.

  • TimeFilter: The results on Solar-Energy, ECL, and Traffic were taken from the original paper. For the other datasets, we reran the dataset-specific scripts under scripts/ in the official repository, retaining the default model-specific settings and applying the common training settings above.

  • OLinear: The original paper uses a weighted L1 loss. For a consistent training objective, we used MSE loss and reran the corresponding dataset scripts under scripts/OLinear/ in the official repository, retaining the default model-specific settings and applying the common training settings above.

  • VPNet: The results on Solar-Energy, ECL, and Traffic were taken from the original paper. For the PEMS datasets not evaluated in the original paper or official repository, we reran VPNet using the common training settings above and MSE loss. The key model settings were hidden_len=256, patch length 16, and kernel_vars=17.

  • TimeXer: The results on ECL and Traffic were taken from the original paper. For the PEMS datasets not evaluated in the original paper or official repository, we reran TimeXer using the common training settings above and MSE loss, with patch_len=16 and d_model=512.

  • TimePro: The results on Solar-Energy and ECL were taken from the original paper. For the PEMS datasets not evaluated in the original paper or official repository, we reran TimePro with patch_len=8, stride=4, e_layers=2, and a learning rate of 0.0005. The maximum training epochs, early-stopping patience, and dropout followed the common settings above.

Reviewer_dUbxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TimeRelay for multivariate time series forecasting. Its main component, Relation-Aware State Propagation (RASP), uses a compact multivariate context to control recurrent information propagation across variables, avoiding explicit pairwise interactions. The model also uses multi-scale temporal embeddings and parallel prefix scan for efficient computation. Experiments are conducted on ten datasets to evaluate forecasting accuracy, scalability, and the effectiveness of the proposed components.

Critical strengths
  1. The paper is generally well organized, and the proposed method is clearly explained.

  2. TimeRelay achieves competitive forecasting results on several benchmarks. The authors also provide comparisons with different cross-variable modeling methods, along with ablation and efficiency studies.

Critical weaknesses
  1. RASP uses an aggregated representation of the other variables as context. This may lose information about specific variable relationships, especially when only a small subset of variables is relevant. The paper does not examine this limitation.

  2. The recurrent propagation depends on the order of variables, but there is no natural ordering among them. The appendix tests several fixed orders, with separate training for each. It would be more useful to examine how a trained model behaves when the variable order changes.

  3. The novelty of RASP is not entirely clear. Existing methods already use compact global representations to exchange information across variables. The paper does not sufficiently explain what RASP adds beyond these approaches.

  4. Table 4 compares several cross-variable mixers, but does not include a simple global-context mixer without recurrence. Such a comparison would help determine whether recurrent propagation is really needed.

  5. Tables 1 and 2 only report results averaged over forecasting horizons. The results at each horizon are also important.

  6. Eight of the ten datasets are traffic-related. Despite the number of benchmarks, the experiments cover only a few application domains, making the claimed generality less convincing.

Questions for authors
  1. How does the aggregated context preserve dependencies between specific variable pairs, particularly when the relevant relationships are sparse?

  2. What is the main advantage of recurrent state propagation over a simpler global-context mixing mechanism?

  3. How sensitive is a trained TimeRelay model to changes in variable ordering?

  4. Could the authors provide the forecasting results for each prediction horizon?

6 comments
AuthorsOpenReview ↗

Response to Reviewer dUbx (Part I)

Thank you for your constructive comments. We have addressed all your comments below with additional experiments.


W1 & Q1. The role of aggregated context in RASP.

Our RASP. The aggregated context in RASP conditions state transitions; it is not the sole channel for transmitting cross-variable information. Our formulation distinguishes identifying pairwise dependencies from propagating predictive information. RASP focuses on the latter and does not require the context to reconstruct a complete variable-to-variable interaction structure.

Specifically, each variable separately generates its candidate representation in Eq. (11), which enters the recurrent state through Eq. (12). Cross-variable information is therefore propagated through individual state updates, not only through the aggregated context. The gate determines how much of the propagated state is retained and how much of the current candidate is incorporated. The context is also not simply a fixed mean. Eq. (9) applies learnable transformations before aggregation and residual refinement afterward, allowing the context representation to be learned for the forecasting objective. In Eq. (10), each gate is conditioned on the current variable and its relation to the leave-one-out multivariate context. These mechanisms allow information contributions to vary across variables and input windows.

Absolute Pearson correlations. To examine performance across different correlation regimes, we computed absolute Pearson correlations between complete variable time series over the training period. Table 6 below reports the mean, median, and proportions of variable pairs at different correlation thresholds, using all unique variable pairs and excluding self-correlations.

Table 6. Pairwise correlation statistics over the training period.

DatasetVariablesUnique pairsMeanMedianWeak pairs (|r| < 0.2)|r| < 0.5|r| ≥ 0.5Strong pairs (|r| ≥ 0.8)
PEMS0335863,9030.8420.8590.00%0.24%99.76%73.43%
PEMS0430746,9710.7700.8000.30%4.59%95.41%50.13%
PEMS07883389,4030.8050.8290.25%1.87%98.13%61.12%
PEMS0817014,3650.7880.8160.36%3.98%96.02%54.91%
PEMS-BAY32552,6500.3640.34735.54%65.95%34.05%4.33%
METR-LA20721,3210.6030.6000.07%21.59%78.41%7.68%
Solar-Energy1379,3160.9140.9140.00%0.00%100.00%99.99%
ECL32151,3600.4970.47517.29%52.59%47.41%21.98%
Traffic862371,0910.5790.6083.57%30.32%69.68%9.16%

Forecasting results. Based on mean absolute correlation, we selected the two lowest-correlation datasets, PEMS-BAY and ECL, and the two highest-correlation datasets, Solar-Energy and PEMS03. We also included METR-LA as an intermediate-correlation example. Table 7 below reports their forecasting results with RASP, None, PlainScan, and Attention under the same temporal embedding and prediction head. Note that the datasets were selected by correlation statistics, not forecasting performance.

RASP achieves lower or tied MSE and lower MAE than all three alternatives on these five datasets. Its improvement over None supports the predictive value of cross-variable modeling even when overall correlations are weaker. The comparison with PlainScan supports the benefit of variable–context conditioning, while the comparison with Attention supports the effectiveness of compact context and state-mediated propagation within the shared backbone. These results support RASP's effectiveness across different correlation regimes, including datasets with weaker overall pairwise correlations.

Table 7. Forecasting performance across lower, moderate, and higher overall pairwise correlations.

MethodMetricPEMS-BAY (lower)ECL (lower)METR-LA (moderate)Solar-Energy (higher)PEMS03 (higher)
TimeRelay (RASP)MSE0.3620.1670.7160.1920.090
MAE0.2770.2610.4740.2490.197
+ NoneMSE0.4470.1820.7340.2120.120
MAE0.3200.2700.4850.2620.224
+ PlainScanMSE0.3690.1670.7430.1970.092
MAE0.2840.2630.4870.2510.198
+ AttentionMSE0.3670.1670.7640.2050.093
MAE0.2810.2630.4900.2670.199
AuthorsOpenReview ↗

Response to Reviewer dUbx (Part II)


W2 & Q3. Sensitivity to test-time changes in variable ordering

RASP traverses variables in a fixed order to propagate cross-variable information, without assuming that this order has an intrinsic semantic meaning. Bidirectional propagation combines information from both traversal directions, but does not guarantee permutation equivariance. The experiment in Appendix F evaluates learning under different fixed traversals, with a separately trained model for each order and the same order maintained throughout training, validation, and testing.

Evaluation. To directly evaluate this sensitivity, we kept the trained model parameters fixed and changed the variable order only at test time, without retraining or fine-tuning. We evaluated partial permutations with ratios of 0.25, 0.5, and 0.75, a full random permutation, and reverse order. For each partial permutation, we randomly selected approximately the specified fraction of variables and swapped them in pairs, leaving all other variables in their original positions. Each permutation was fixed across the test set. Predictions were restored to the original variable order before evaluation, ensuring alignment with the ground truth.

Results. The reults in Table 8 below shows that a trained TimeRelay model is sensitive to test-time changes in variable ordering. Both MSE and MAE increase on all datasets, and errors increase with the partial-permutation ratio in all datasets. The magnitude of degradation varies across datasets. At a ratio of 0.25, MSE increases from 0.090 to 0.113 on PEMS03 and from 0.167 to 0.249 on ECL, compared with 0.716 to 0.734 on METR-LA. Under random order, MSE reaches 0.356 on ECL and 0.667 on PEMS-BAY, while METR-LA reaches 0.777. Random and reverse orders produce broadly similar degradation.

These results distinguish learning under different fixed traversals from robustness to changing the traversal after training. Although a natural semantic order is not required, consistency between the training and inference traversals matters. In RASP, changing the traversal changes the sequence of state updates and the composition of propagated information.

Table 8. Sensitivity of a trained TimeRelay model to test-time changes in variable ordering.

Variable orderMetricPEMS03PEMS04PEMS07PEMS08PEMSD7ECLTrafficSolarPEMS-BAYMETR-LA
TimeRelay (original order)MSE0.0900.0880.0730.1140.3240.1670.4270.1920.3620.716
MAE0.1970.1950.1680.2010.3130.2610.2810.2490.2770.474
Test-time partial order (0.25)MSE0.1130.1130.1010.1420.3780.2490.4480.2010.4370.734
MAE0.2170.2190.1980.2270.3440.3190.2960.2550.3100.479
Test-time partial order (0.5)MSE0.1360.1430.1290.1530.4490.2810.4690.2440.5230.749
MAE0.2400.2460.2290.2430.3800.3430.3090.2810.3410.488
Test-time partial order (0.75)MSE0.1570.1700.1520.1710.5030.3170.4910.2500.6030.759
MAE0.2600.2700.2550.2620.4100.3680.3220.2850.3700.493
Test-time random orderMSE0.1740.2020.1790.2000.6030.3560.5290.2370.6670.777
MAE0.2800.2980.2850.2860.4590.3920.3390.2770.3980.505
Test-time reverse orderMSE0.1780.2000.1760.2120.6010.3770.5300.2380.6720.775
MAE0.2820.2960.2820.2890.4620.4050.3390.2770.3960.502
AuthorsOpenReview ↗

Response to Reviewer dUbx (Part III)


W3. What RASP adds beyond compact global representations

RASP's contribution lies not in using a compact global representation alone, but in how multivariate context controls cross-variable information propagation. For example, Attention exchanges information through explicit pairwise attention weights, while SOFTS's STAR aggregates variable representations into a shared core and redistributes it through MLP fusion. RASP separates context conditioning from state-mediated information propagation: the compact context conditions each state transition, while the recurrent state carries predictive information across individual variables.

Specifically, RASP constructs a leave-one-out context with learnable residual refinement in Eqs. (8)–(9). In Eq. (10), each gate is conditioned on the current variable and its relation to this context. The gate adaptively determines how much propagated predictive information is retained and how much of the current variable's candidate representation is incorporated in Eqs. (11)–(12). This enables information contributions to vary across variables and input windows without explicitly computing all variable-to-variable scores. Both the gate and candidate are independent of the previous recurrent state, preserving the affine structure for efficient parallel evaluation through associative scan.

Performance comparison. The module-replacement comparison in Table 2 (Response to Reviewer rjqd) provides empirical support. With the same temporal embedding and prediction head, RASP achieves the lowest average MSE and MAE among RASP and the adapted cross-variable modules from SOFTS, TQNet, TimePro, VPNet, and minGRU. These results support the effectiveness of RASP's cross-variable modeling mechanism within the shared forecasting backbone.


W4 & Q2. The benefit of recurrent state propagation beyond global-context mixing

The main advantage of RASP is that recurrent states carry predictive information accumulated through individual variable updates, providing a propagation channel beyond the aggregated context. The relation-aware gate adaptively determines how much propagated information is retained and how much of the current variable's candidate is incorporated (Eqs. (10)–(12)).

To examine this benefit, we evaluated a parameter-matched global-context mixing variant without recurrence. We retained the same leave-one-out context encoder, relation features, gates, candidate projections, output projection, and residual connection, together with the same temporal embedding and prediction head. Only recurrent state propagation was removed.

Specifically, RASP computes the forward update as hi→=gi→⊙hi−1→+(1−gi→)⊙ui→h_i^{\rightarrow} = g_i^{\rightarrow} \odot h_{i-1}^{\rightarrow} + (1-g_i^{\rightarrow}) \odot u_i^{\rightarrow}, whereas the nonrecurrent variant computes hi→=(1−gi→)⊙ui→h_i^{\rightarrow} = (1-g_i^{\rightarrow}) \odot u_i^{\rightarrow}. The backward branch is modified in the same way. Thus, each variable still receives a relation-conditioned update through the aggregated context, but no recurrent state is propagated between variables.

Results. Table 9 below reports the comparison across all datasets. RASP achieves lower MSE on all ten datasets and lower MAE on nine, with a tie on PEMS07. For example, it reduces MSE from 0.215 to 0.192 on Solar, from 0.439 to 0.427 on Traffic, and from 0.723 to 0.716 on METR-LA. With the context encoder and relation features fixed, these results support the additional predictive benefit of recurrent state propagation over global-context mixing without recurrence within the shared forecasting backbone.

Table 9. Comparison of RASP with global-context mixing without recurrent propagation.

MethodMetricPEMS03PEMS04PEMS07PEMS08PEMSD7ECLTrafficSolarPEMS-BAYMETR-LA
TimeRelay (RASP)MSE0.0900.0880.0730.1140.3240.1670.4270.1920.3620.716
MAE0.1970.1950.1680.2010.3130.2610.2810.2490.2770.474
+ Global-context mixing w/o recurrenceMSE0.0960.0910.0740.1160.3260.1690.4390.2150.3700.723
MAE0.2030.2000.1680.2030.3150.2650.2860.2640.2800.483
Reviewer_rjqdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TimeRelay, a channel-dependent forecaster for multivariate time series that first encodes each variable independently (linear projection plus a multi-scale patch-MLP residual). Cross-variable information is then exchanged by Relation-Aware State Propagation (RASP), a bidirectional gated linear recurrence along the variable axis whose gate depends only on each variable's relation to a leave-one-out mean context. Because neither gate nor candidate depends on h_{i-1}, the recurrence is affine and evaluated with a Blelloch-style parallel scan (O(C) work, O(log C) depth). The motivation is that explicit pairwise interaction scales quadratically in C and that pairwise correlations are heterogeneous. Experiments cover 10 benchmarks at L = 96 against 14 baselines, with mixer-replacement, context, traversal-order and efficiency studies.

Critical strengths
  1. Table 4 swaps only the cross-variable mixer (None, MLP, Attention, Mamba, BiGRU, Context-BiGRU, PlainScan) while keeping encoder and head fixed, and the appendix further ablates context, gate terms, traversal order and direction. This is the right way to attribute gains to the proposed module, and some findings are informative on their own, e.g., on METR-LA the no-mixing variant beats Attention, Mamba and BiGRU.
  2. RASP is simple, has O(C) work and low memory (Table 8: about 2.7 GB vs. 11.4 GB for attention on PEMS07 at D = 64), and is insensitive to variable ordering (Table 10). This makes it a practical drop-in cross-variable block, as also suggested by the transfer experiment in Appendix D.3.
Critical weaknesses
  1. The motivation is weak and novelty is limited. With C <= 883, per-epoch runtime of Attention vs. RASP is 32/33/41 s vs. 31/30/33 s (Table 8), memory-efficient attention kernels avoid the C x C matrix, and Attention ties RASP on ECL and PEMS07 (within 0.003 MSE on Traffic). The heterogeneity argument (Fig. 1a) has no counterpart in the method, and close relatives (SOFTS's STAR, TQNet, S-Mamba/TimePro, VPNet, and the minGRU / gated linear RNN family) are not discussed; RASP amounts to a bidirectional minGRU along the variable axis with global-context gate features.
  2. Gains are small relative to uncertainty and several numbers are inconsistent. Against the best alternative mixer in Table 4, RASP improves MSE by 0.005 on Solar, 0.018 on METR-LA (vs. no mixing), 0.003 on Traffic and 0 on ECL/PEMS07, with no seeds or std reported. In Fig. 5 no depth reaches the main results (e.g., PEMS-BAY MSE 0.379-0.394 vs. 0.362), and several baseline entries in Tables 1-2 appear duplicated or below published values; using published numbers would remove TimeRelay's first place on several metrics.
Questions for authors
  1. Why is RASP needed, given that attention has similar runtime and accuracy at C <= 883? This deserves further analysis.
6 comments
AuthorsOpenReview ↗

Response to Reviewer rjqd (Part I)

Thank you for your constructive comments. We have addressed all your comments below with additional experiments.


W1.1 & Q1. Motivation and comparison with Attention

Runtime. Attention and RASP have comparable epoch times at the tested scales. Dense attention can exploit highly parallel GPU matrix multiplications, so computing the C×CC \times C interactions need not incur a large runtime penalty when C≤883C \leq 883. Thus, the runtime improvements in Table 8 of the original paper are modest, and a large speedup is not the sole motivation for RASP.

Memory relative to explicit Attention. Table 8 of the original paper nevertheless shows substantially lower GPU memory usage for RASP than for the explicit Attention implementation: approximately 43%, 73%, and 76% reductions on ECL, Traffic, and PEMS07, respectively, at D=64D=64. Explicit Attention materializes attention scores and weights of size B×H×C×CB \times H \times C \times C, which becomes costly as the number of variables increases. RASP constructs compact contexts and propagates predictive information through recurrent states without forming these pairwise matrices.

Comparison with FlashAttention. To account for memory-efficient attention kernels, we additionally compared RASP with FlashAttention, with the Attention and FlashAttention modules aligned in architecture and all three methods using the same backbone and four model layers. To accommodate FlashAttention's FP16/BF16 requirements and ensure a consistent precision policy, we used AMP with FP16 autocasting for all three methods. The results are reported in Table 15 below of our response to Reviewer m8RB. At D = 64, FlashAttention and RASP have very similar GPU memory usage across the three datasets. FlashAttention avoids storing the full attention matrices, requiring linear intermediate storage in C at a fixed batch size and D, but retains dense pairwise computation. RASP instead uses context-conditioned state propagation with linear arithmetic work in C at a fixed width, while maintaining additional D-dependent components for context construction, gates, and candidates. Despite their similar memory usage at D = 64, RASP achieves lower MSE and MAE on all three datasets.

Forecasting performance. We further report the forecasting comparison across all datasets in Table 1 below, keeping the same temporal embedding and prediction head and replacing only the cross-variable mixer. RASP achieves lower MSE on nine datasets and ties Attention on ECL, while achieving lower MAE on all ten datasets. Average MSE and MAE decrease by 3.48% and 2.75%, respectively. These results support the effectiveness of relation-conditioned state propagation compared with explicit pairwise attention within the evaluated forecasting backbone.

Table 1. Average performance comparison of RASP and Attention across all datasets.

MethodMetricPEMS03PEMS04PEMS07PEMS08PEMSD7ECLTrafficSolarPEMS-BAYMETR-LAAverageImprovement
TimeRelay (RASP)MSE0.0900.0880.0730.1140.3240.1670.4270.1920.3620.7160.255—
MAE0.1970.1950.1680.2010.3130.2610.2810.2490.2770.4740.262—
+ AttentionMSE0.0930.0910.0740.1190.3350.1670.4300.2050.3670.7640.265+0.009 (3.48%)
MAE0.1990.1990.1730.2080.3220.2630.2880.2670.2810.4900.269+0.007 (2.75%)

Motivation and mechanism. RASP is motivated by the need to adaptively use cross-variable information for more accurate forecasting while improving computational efficiency by avoiding explicit computation of all pairwise interactions. As illustrated in Fig. 1a, correlation strengths vary across variable pairs, and each variable may benefit differently from the remaining multivariate system. RASP addresses this through variable–context-conditioned transitions: Eq. (9) constructs a learnable leave-one-out context, and Eq. (10) uses the current variable, its context, and their relation features to determine how much propagated state is retained and how much of the current candidate is incorporated. This allows information contributions to adapt across variables and input windows. The updates preserve an affine structure, enabling efficient parallel evaluation through associative scan with linear arithmetic work in CC at a fixed width.

AuthorsOpenReview ↗

Response to Reviewer rjqd (Part II)


W1.2. Connection between heterogeneity and the method

Heterogeneity. Fig. 1a illustrates non-uniform pairwise correlations: correlation strengths vary substantially among variable pairs, and correlation distributions differ across datasets. This motivates a modeling approach that uses the current variable's features and its relation to the context of other variables to adaptively determine how cross-variable information contributes to its representation.

Our RASP. RASP implements this adaptation through relation-aware recurrent state transitions conditioned on compact multivariate context. In Eq. (10), the current variable, its leave-one-out context, their absolute difference, and their element-wise interaction provide complementary relational information for computing the feature-wise gate. The gate can therefore vary across variables and input windows. In Eq. (12), it determines how much information is retained from the propagated state and how much of the current variable's candidate representation is incorporated. Thus, the variable–context relation modulates the local state transition, while the evolving recurrent state propagates information across variables, without explicitly reconstructing a complete variable-to-variable interaction structure.

Ablations. The ablations in our paper support this design. Table 4 supports the benefit of relation-aware context conditioning beyond affine recurrent propagation alone. Table 6 shows that multivariate context and its learnable residual refinement provide useful conditioning signals. Table 7 supports the complementary contributions of variable–context relation terms to the gate. These results support the mechanism motivated by heterogeneity: adapting state transitions according to each variable's features and its relation to the multivariate context.


W1.3. Closely related methods

Other methods. SOFTS's STAR aggregates variable representations into a shared core and redistributes it through MLP fusion. TQNet uses periodically shifted learnable queries and input-derived keys and values for cross-variable attention. S-Mamba models variables through bidirectional selective SSM scans, while TimePro introduces time-tune to construct hyper-states. VPNet uses depthwise convolutions over local variable–patch neighborhoods. minGRU uses input-dependent gates and an affine recurrence that permits parallel evaluation.

Our contribution. RASP shares the affine recurrence foundation with minGRU. Our contribution is relation-aware conditioning of state propagation to use contextual information from other variables to enhance the current variable's representation, while preserving the affine structure for efficient parallel evaluation through associative scan. In Eq. (10), each gate is conditioned on the current variable and its relation to the leave-one-out multivariate context. The context conditions the state transition through the gate, while the recurrent state propagates predictive information across variables. Both the gate and candidate are independent of the previous state, preserving the affine recurrence required for associative scan.

Experiments. We also conducted module-level comparisons by replacing RASP with adapted cross-variable modules while retaining the same temporal embedding and prediction head. These include SOFTS (STAR), TQNet (embedding-space TQ-MHA), TimePro (variable-axis selective SSM without time-tune), VPNet (variable-only VarTCN), and minGRU (using log-space evaluation, extended to bidirectional propagation along the variable axis with averaged directional outputs). This setup compares cross-variable modeling approaches within the same forecasting backbone. "+ Method" denotes replacing RASP with an adapted cross-variable module from the corresponding method, rather than adding it on top of RASP or reproducing the full original model.

As shown in Table 2 below, RASP achieves the lowest average MSE and MAE across the ten datasets. Compared with the adapted modules from SOFTS, TQNet, TimePro, VPNet, and minGRU, it reduces average MSE by 0.013, 0.026, 0.015, 0.018, and 0.019, respectively, and average MAE by 0.008, 0.017, 0.011, 0.010, and 0.019. These results support the effectiveness of RASP as a cross-variable modeling mechanism within the shared forecasting backbone.

AuthorsOpenReview ↗

Response to Reviewer rjqd (Part III)

Table 2. Comparison of RASP with other cross-variable modeling modules.

MethodMetricPEMS03PEMS04PEMS07PEMS08PEMSD7ECLTrafficSolarPEMS-BAYMETR-LAAverageImprovement
TimeRelayMSE0.0900.0880.0730.1140.3240.1670.4270.1920.3620.7160.255—
MAE0.1970.1950.1680.2010.3130.2610.2810.2490.2770.4740.262—
+SOFTSMSE0.0940.0960.0780.1160.3260.1880.4790.1980.3640.7470.269+0.013
MAE0.2020.2080.1800.2120.3130.2840.3170.2450.2760.4620.270+0.008
+TQNetMSE0.1130.0940.0810.2510.3330.1680.4700.2030.3750.7250.281+0.026
MAE0.2120.2040.1790.2780.3240.2620.2910.2690.2890.4790.279+0.017
+TimeProMSE0.0930.0930.0730.1410.3300.1710.4510.2030.4170.7300.270+0.015
MAE0.2010.2010.1680.2260.3210.2640.2840.2580.3080.4910.272+0.011
+VPNetMSE0.0960.0930.0770.1250.3650.1750.4390.2060.4170.7390.273+0.018
MAE0.2040.2030.1760.2130.3400.2690.2850.2590.2950.4730.272+0.010
+minGRUMSE0.0980.0970.1060.1290.3380.1890.4620.1960.3720.7510.274+0.019
MAE0.2070.2090.2110.2250.3280.2900.2970.2470.2860.5080.281+0.019

W2.1. Gains and uncertainty

Different seeds. To quantify variability across random seeds, we evaluated RASP and several alternative mixers using seeds 2024, 2025, and 2026. We averaged the four prediction horizons within each seed and report the mean and standard deviation across seeds in Table 3 below. RASP achieves the lowest or tied-lowest mean MSE on all five datasets and the lowest or tied-lowest mean MAE on four. On METR-LA, RASP achieves an MSE of 0.718 ± 0.003, compared with 0.765 ± 0.003 for Attention, 0.747 ± 0.004 for PlainScan, and 0.735 ± 0.005 for Context-BiGRU. On Traffic, RASP achieves the lowest mean MSE, while Context-BiGRU achieves lower mean MAE. On PEMS07, RASP and Context-BiGRU have tied mean MSE and MAE at the reported precision. These results quantify variability across random seeds and show that RASP achieves the best average forecasting performance among the compared mixers across datasets with diverse cross-variable relationships.

Table 3. Comparison of RASP and alternative mixers across random seeds (mean ± std).

MethodMetricSolarMETR-LAECLTrafficPEMS07
TimeRelay (RASP)MSE0.194 ± 0.0020.718 ± 0.0030.166 ± 0.0010.428 ± 0.0010.073 ± 0.001
MAE0.248 ± 0.0010.476 ± 0.0020.261 ± 0.0000.282 ± 0.0010.168 ± 0.000
+ AttentionMSE0.203 ± 0.0040.765 ± 0.0030.168 ± 0.0020.431 ± 0.0010.074 ± 0.001
MAE0.265 ± 0.0050.493 ± 0.0070.264 ± 0.0010.288 ± 0.0000.172 ± 0.001
+ PlainScanMSE0.197 ± 0.0030.747 ± 0.0040.167 ± 0.0000.442 ± 0.0040.074 ± 0.000
MAE0.250 ± 0.0010.491 ± 0.0060.263 ± 0.0000.286 ± 0.0000.172 ± 0.000
+ Context-BiGRUMSE0.204 ± 0.0010.735 ± 0.0050.172 ± 0.0030.436 ± 0.0010.073 ± 0.000
MAE0.258 ± 0.0030.479 ± 0.0030.266 ± 0.0020.279 ± 0.0000.168 ± 0.000

More datasets. To assess whether RASP's gains extend beyond the five datasets evaluated in the original paper's Table 4, we expanded the alternative-mixer comparisons to all datasets, retaining the same temporal embedding and prediction head. As shown in Table 4 below, RASP achieves the lowest average MSE and MAE among the compared mixers. Compared with the strongest alternative for each metric, RASP reduces average MSE by 0.007 (2.71%) relative to PlainScan and average MAE by 0.004 (1.47%) relative to Context-BiGRU. These results support the effectiveness of RASP across a broader set of datasets within the shared forecasting backbone.

Comments on the paper

1 comment

Dear Reviewers rjqd, dUbx, and m8RB,

The authors have responded to your questions. Please review their responses and consider whether your concerns have been addressed. You may also read the other reviewers' comments and the revised manuscript, and share any further feedback.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.