- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TimeRelay combines multiscale temporal embeddings with a leave-one-out context that controls bidirectional propagation across variables. Its gates and candidate updates are independent of the previous recurrent state, allowing exact evaluation through an associative affine scan. The paper evaluates forecasting accuracy, alternative mixers, computational scaling and traversal choices on ten datasets.
Critical strengths
-
The recurrence and its parallel evaluation fit together cleanly. Associative composition changes the execution schedule rather than approximating the model. Table 9 compares implementations of the same recurrence and reports a PEMS07 epoch-time reduction from 350 to 33 seconds.
-
The evaluation contains useful controlled comparisons, not just complete-model rankings. Tables 3–4 separate temporal encoding, context and alternative mixers; the direction and plug-in experiments provide further support. The reported gains are strongest on traffic datasets and Solar, rather than uniformly across all long-horizon benchmarks.
Critical weaknesses
-
The evidence does not fully separate global contextualization from recurrent propagation. PlainScan tests recurrence without relation-aware context, and Context-BiGRU tests a different recurrence with context. Neither is a context-conditioned, nonrecurrent block with the same encoder, relation features and output budget. That control would directly test what propagation adds after the other variables have already been summarized. Table 4 also ties RASP with Mamba and Context-BiGRU on PEMS07, while Table 6 sometimes favors mean context. These are useful results, but Section 5.4's exclusion of context or recurrence alone as explanations is stronger than the controls establish. Robustness to separately trained traversal orders does not resolve this attribution question.
-
The accuracy and efficiency evidence needs a clearer connection to the evaluated configurations. The D=64 comparison is a legitimate equal-width control, but main forecasting runs use D=512; Tables 8–9 therefore do not establish the main model's accuracy–cost trade-off. The attention backend and measurement protocol should be reported, with a fused SDPA/FlashAttention comparison where supported [1]. The paper also omits per-horizon results and variability for close comparisons, and Figure 8 does not specify its forecast horizon or aggregation. Please provide the corresponding run/configuration records, including baseline provenance and the patience-3 versus released patience-5 setting, rather than leaving these details implicit.
References
[1] Dao et al. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. 2022. https://arxiv.org/abs/2205.14135
Questions for authors
-
What happens when the leave-one-out context and relation-conditioned features are retained but recurrent propagation is replaced by a parameter-matched nonrecurrent update?
-
Can you provide horizon-level results, the settings behind Figure 8, and representative D=512 cost measurements? Please identify the attention backend, actual training configurations and which baseline entries were rerun.
5 comments
Response to Reviewer m8RB (Part II)
W2.3 & Q2.3. Attention backend, measurement protocol, and comparison with fused attention.
To connect the efficiency comparison to the main forecasting results in Tables 1–2 of the original paper, we evaluated Attention, FlashAttention, and RASP at D = 512 on ECL, Traffic, and PEMS07, using four model layers for all three methods. We also evaluated D = 64, 128, and 256 while keeping the layer count fixed at four to examine how the accuracy–cost trade-off changes with embedding width.
Attention and FlashAttention backend. The Attention baseline uses an explicit implementation that computes , applies softmax, and multiplies the resulting attention weights by . The FlashAttention baseline uses PyTorch's scaled_dot_product_attention with the FLASH_ATTENTION backend explicitly enabled. Its normalization, residual connections, and dropout settings are aligned with the Attention baseline. The CUDA FlashAttention backend used in our implementation requires FP16 or BF16 Q/K/V tensors. We therefore used CUDA automatic mixed precision (AMP) with FP16 autocasting for all three methods to ensure a consistent precision policy. All experiments were conducted on an NVIDIA H20 GPU with a batch size of 32. We retained the same temporal embedding and prediction head and replaced only the cross-variable mixer. We report the number of trainable parameters, training time per epoch, and GPU memory usage.
Time cost. Table 15 below shows that RASP achieves the lowest or tied-lowest epoch time across all tested datasets and embedding dimensions, although the runtime improvements are modest. At D = 512, the epoch times for Attention, FlashAttention, and RASP are 32/32/31 s on ECL, 30/29/28 s on Traffic, and 38/33/32 s on PEMS07. Attention benefits from highly parallel GPU matrix operations, while FlashAttention further reduces memory traffic through fused kernels. These advantages may keep attention fast at the tested variable counts despite its dense pairwise computation. Since epoch time includes the shared temporal embedding and prediction head, mixer-level runtime differences may be less pronounced.
Memory cost. FlashAttention uses less GPU memory than RASP, while RASP uses substantially less memory than explicit Attention. At D = 64, FlashAttention and RASP have similar memory usage across the three datasets despite their different information-exchange mechanisms. FlashAttention achieves this through tiled, fused computation and recomputation during backpropagation, avoiding storage of the full attention matrices. Its attention intermediates require linear storage in C at a fixed batch size and embedding dimension D, although it retains dense pairwise computation. RASP's intermediate storage also scales linearly with C under these fixed conditions, but its memory gap relative to FlashAttention widens as D increases. RASP maintains additional D-dimensional representations for context construction, relation-aware gates, candidates, and bidirectional propagation, which may contribute to its higher memory usage at larger D. In future work, we will explore optimizations along both the variable and embedding dimensions to reduce the dependence of memory usage on D and further lower RASP's linear memory cost in C.
Performance. RASP achieves the lowest or tied-lowest MSE and the lowest MAE across all tested settings. The gains vary across datasets and dimensions: at D = 64, RASP reduces MSE from 0.096 to 0.085 on PEMS07 and from 0.490 to 0.478 on Traffic compared with both attention baselines. At D = 512, it reduces MSE from 0.431 to 0.429 on Traffic and from 0.076 to 0.075 on PEMS07. On ECL, MSE is tied at 0.167, while MAE decreases from 0.263 to 0.262. These results characterize RASP's accuracy–cost trade-off on the evaluated datasets: competitive training time, substantially lower GPU memory usage than explicit Attention, memory usage close to FlashAttention at D = 64, and improved forecasting accuracy.
Response to Reviewer m8RB (Part III)
Table 15. Comparison of Attention, FlashAttention, and RASP across embedding dimensions.
| D | Dataset | Metric | Attention | FlashAttention | RASP | Dataset | Attention | FlashAttention | RASP | Dataset | Attention | FlashAttention | RASP |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 512 | ECL | Parameters (M) | 5.120 | 5.120 | 9.333 | Traffic | 5.448 | 5.448 | 9.661 | PEMS07 | 5.461 | 5.461 | 9.674 |
| Time (s) | 32 | 32 | 31 | 30 | 29 | 28 | 38 | 33 | 32 | ||||
| Memory (MB) | 2815 | 1887 | 2347 | 10739 | 4095 | 4797 | 11169 | 4159 | 4893 | ||||
| MSE | 0.167 | 0.167 | 0.167 | 0.431 | 0.431 | 0.429 | 0.076 | 0.076 | 0.075 | ||||
| MAE | 0.263 | 0.263 | 0.262 | 0.287 | 0.287 | 0.282 | 0.172 | 0.172 | 0.170 | ||||
| 256 | ECL | Parameters (M) | 1.414 | 1.414 | 2.472 | Traffic | 1.604 | 1.604 | 2.662 | PEMS07 | 1.612 | 1.612 | 2.669 |
| Time (s) | 31 | 30 | 30 | 29 | 28 | 27 | 35 | 31 | 31 | ||||
| Memory (MB) | 2471 | 1545 | 1767 | 10287 | 3425 | 3845 | 9927 | 3377 | 3797 | ||||
| MSE | 0.168 | 0.168 | 0.167 | 0.443 | 0.443 | 0.438 | 0.078 | 0.078 | 0.076 | ||||
| MAE | 0.263 | 0.263 | 0.262 | 0.297 | 0.297 | 0.290 | 0.179 | 0.179 | 0.172 | ||||
| 128 | ECL | Parameters (M) | 0.444 | 0.444 | 0.711 | Traffic | 0.564 | 0.564 | 0.831 | PEMS07 | 0.569 | 0.569 | 0.835 |
| Time (s) | 31 | 30 | 30 | 28 | 28 | 27 | 35 | 31 | 30 | ||||
| Memory (MB) | 2357 | 1521 | 1623 | 9733 | 3173 | 3327 | 10127 | 3193 | 3349 | ||||
| MSE | 0.175 | 0.175 | 0.173 | 0.464 | 0.464 | 0.453 | 0.084 | 0.084 | 0.081 | ||||
| MAE | 0.268 | 0.268 | 0.267 | 0.313 | 0.313 | 0.305 | 0.187 | 0.187 | 0.181 | ||||
| 64 | ECL | Parameters (M) | 0.191 | 0.191 | 0.259 | Traffic | 0.286 | 0.286 | 0.354 | PEMS07 | 0.290 | 0.290 | 0.358 |
| Time (s) | 30 | 30 | 29 | 28 | 28 | 26 | 33 | 31 | 30 | ||||
| Memory (MB) | 2409 | 1461 | 1477 | 9625 | 2945 | 3031 | 10019 | 2977 | 3055 | ||||
| MSE | 0.183 | 0.183 | 0.182 | 0.490 | 0.490 | 0.478 | 0.096 | 0.096 | 0.085 | ||||
| MAE | 0.275 | 0.275 | 0.273 | 0.333 | 0.333 | 0.327 | 0.202 | 0.202 | 0.188 |
W2.4 & Q2.4. Training configurations, baseline provenance, and rerun settings.
The training configurations of TimeRelay are described in Appendix B.2, Implementation Details. The released code uses an early-stopping patience of 5. For the baseline experiments we reran, we used a maximum of 15 training epochs, an early-stopping patience of 5, learning rate of 0.001, dropout of 0.1, hidden dimension of 512, and batch size of 32, unless otherwise specified below. We distinguish results taken from the original papers from those obtained through our reruns as follows.
-
iTransformer, SOFTS, and TQNet: We used the results reported in their original papers.
-
CycleNet: We reran the CycleNet/MLP variant using the dataset-specific scripts under scripts/CycleNet in the official repository. We retained the default model-specific settings and applied the common training settings above.
-
TimeFilter: The results on Solar-Energy, ECL, and Traffic were taken from the original paper. For the other datasets, we reran the dataset-specific scripts under scripts/ in the official repository, retaining the default model-specific settings and applying the common training settings above.
-
OLinear: The original paper uses a weighted L1 loss. For a consistent training objective, we used MSE loss and reran the corresponding dataset scripts under scripts/OLinear/ in the official repository, retaining the default model-specific settings and applying the common training settings above.
-
VPNet: The results on Solar-Energy, ECL, and Traffic were taken from the original paper. For the PEMS datasets not evaluated in the original paper or official repository, we reran VPNet using the common training settings above and MSE loss. The key model settings were hidden_len=256, patch length 16, and kernel_vars=17.
-
TimeXer: The results on ECL and Traffic were taken from the original paper. For the PEMS datasets not evaluated in the original paper or official repository, we reran TimeXer using the common training settings above and MSE loss, with patch_len=16 and d_model=512.
-
TimePro: The results on Solar-Energy and ECL were taken from the original paper. For the PEMS datasets not evaluated in the original paper or official repository, we reran TimePro with patch_len=8, stride=4, e_layers=2, and a learning rate of 0.0005. The maximum training epochs, early-stopping patience, and dropout followed the common settings above.
Response to Reviewer m8RB (Part I)
Thank you for your constructive comments. We have addressed all your comments below with additional experiments.
W1 & Q1. Comparison of RASP with global-context mixing without recurrence.
RASP uses compact multivariate context to condition state transitions, while recurrent states carry predictive information accumulated through individual variable updates. The relation-aware gate determines how much propagated information is retained and how much of the current variable's candidate is incorporated.
To directly isolate the contribution of propagation, we evaluated a parameter-matched, context-conditioned nonrecurrent variant. We retained the same leave-one-out context encoder, relation features, gates, candidate projections, output projection, and residual connection, together with the same temporal embedding and prediction head. Only recurrent state propagation was removed.
Specifically, RASP computes the forward update as hi→=gi→⊙hi−1→+(1−gi→)⊙ui→, whereas the nonrecurrent variant computes hi→=(1−gi→)⊙ui→. The backward branch is modified in the same way. Thus, each variable still receives a relation-conditioned update, but no state is propagated between variables.
Results. As shown in Table 13 below, RASP achieves lower MSE on all ten datasets and lower MAE on nine, with a tie on PEMS07. For example, it reduces MSE from 0.215 to 0.192 on Solar, from 0.439 to 0.427 on Traffic, and from 0.723 to 0.716 on METR-LA. Because the nonrecurrent variant retains the same context encoder and relation features, these results support the additional predictive benefit of recurrent state propagation beyond context-conditioned local updates within the shared forecasting backbone.
Table 13. Comparison of RASP with global-context mixing without recurrence.
W2.1 & Q2.1. Horizon-level results
We report TimeRelay's MSE and MAE at each prediction horizon in Table 14 below, together with the unweighted average over the four horizons for each dataset.
Table 14. Forecasting results of TimeRelay at each prediction horizon.
W2.2 & Q2.2. Experimental settings for Fig. 8
In Fig. 8, we vary the input length L∈{96,192,336,576,720} while fixing the forecast horizon at 720 to evaluate forecasting performance over a longer prediction window. The reported results are for this single horizon, not averaged across multiple forecast horizons. Across all input lengths, we use a batch size of 32, a maximum of 15 training epochs, an early-stopping patience of 5, and a learning rate of 0.001.