- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose OREM for test-time adaptation of time-series forecasting after deployment. It adapts through retrieval while keeping the predictor frozen. It sets up two memory banks and also provides this evaluation algorithm. But I think the baselines it compares against are not appropriate.
Critical strengths
- The method does not update any parameters, so it is cheap to implement and very practical.
- The improvement over existing methods is significant.
- It is validated on commonly used time series datasets.
Critical weaknesses
- OneNet and FSNet are the standard references for online adaptation, and RAFT, PFRP and GTR are the retrieval-based forecasters the authors themselves admit are the closest work. There are so many models that should have been compared, but the authors compared none of them.
- The paper does not test multiple seeds. There is no variance, no repeated runs and no statistical analysis of the results, and that is not acceptable.
- The paper never discusses its limitations, and it never analyzes whether retrieval can have negative effects.
Questions for authors
- The authors treat OneNet and FSNet as the standard comparisons for online adaptation, and you say RAFT, PFRP and GTR are the closest retrieval-based forecasters, so why do none of them appear in the experiments? If you do not compare against the most direct competitors, how does your claim of beating existing methods hold up?
- The whole paper runs a single seed, with no variance, no repeated runs and no significance test. For settings that gain less than 1% or as little as 0.225%, why should I believe these are real improvements instead of noise? Can you provide multi-seed results and statistical tests?
- When does retrieval backfire? When the neighbours do not resemble the current window at all, can the correction make the forecast worse? The paper has no failure analysis and no limitations discussion, so can you explain when this method fails?
8 comments
Response to Reviewer hmXe (2/2): W2 / Q2 and W3 / Q3
W2 / Q2
To assess the reproducibility and statistical reliability of the performance improvements, we conducted additional experiments focusing on ETTh2 (2.04%), ETTm2 (2.85%), and Weather (3.47%), which exhibited relatively small average MSE improvements in the original main experiments.
We evaluated 60 configurations, comprising three datasets, five backbones, and four forecast horizons, with three seeds per configuration. For each seed, Frozen and OREM shared exactly the same backbone weights. We calculated the three-seed mean standard deviation (SD) and the two-sided 95% -confidence interval for the seed-wise MSE improvement rate.
Table R4. Multi-seed robustness analysis.
| Dataset | Evaluated | Mean MSE improved | Positive 95% CI lower bound |
|---|---|---|---|
| ETTh2 | 20 | 20 | 17 |
| ETTm2 | 20 | 20 | 20 |
| Weather | 20 | 20 | 20 |
| Overall | 60 | 60 | 57 (95%) |
Positive CI lower bounds refer to the 95% -confidence interval of the seed-wise MSE improvement rate.
In particular, in the 0.225% improvement case highlighted by the reviewer (Weather--TQNet--), all three seeds showed lower MSE. The mean improvement was +0.458% 0.005%, with a 95% CI of [+0.447%, +0.470%].
All 60 configurations reproduced a positive mean improvement, and 57 configurations had 95% confidence intervals entirely above zero. The remaining three configurations also had positive mean improvements, but their confidence intervals included zero. The complete multi-seed results will be incorporated into the revised manuscript.
W3 / Q3
To investigate the potential negative effects and limitations of retrieval, we analyzed the frequency of windows in which OREM increased the actual prediction error and the corresponding average MSE increase.
We define degradation as a test window in which the MSE after adaptation exceeds that of the Frozen backbone, and compare OREM with existing adaptation methods on DLinear and PatchTST for ETTm1, Weather, and Electricity ().
Table R5. Prediction degradation analysis (; lower is better).
| Dataset | Backbone | OREM | TAFAS | PETSA | COSA |
|---|---|---|---|---|---|
| ETTm1 | DLinear | 22.84 / 0.0371 | 33.88 / 0.0418 | 37.16 / 0.0200 | 34.40 / 0.0318 |
| PatchTST | 27.39 / 0.0304 | 39.43 / 0.0103 | 76.86 / 0.0282 | 39.13 / 0.0215 | |
| Weather | DLinear | 19.27 / 0.0080 | 30.02 / 0.0055 | 29.05 / 0.0036 | 34.07 / 0.0039 |
| PatchTST | 27.62 / 0.0081 | 39.76 / 0.0063 | 71.96 / 0.0060 | 36.99 / 0.0037 | |
| Electricity | DLinear | 1.97 / 0.0181 | 24.28 / 0.0032 | 29.14 / 0.0038 | 37.04 / 0.0008 |
| PatchTST | 5.11 / 0.0153 | 23.06 / 0.0041 | 88.29 / 0.0190 | 27.14 / 0.0009 |
Each entry is the degradation rate (%) / the mean MSE increase conditional on degradation.
OREM achieved the lowest window-level MSE degradation rate in all six evaluated configurations. While the conditional magnitude of degradation varied across methods, OREM consistently reduced the frequency of prediction-error increases relative to the frozen backbone. These results provide complementary evidence of OREM's reliability by directly measuring the occurrence of harmful corrections, beyond the direction-based analysis in the original Figure 4. We also examined actual failure cases from six datasets and found that even when input similarity is high, retrieval-based correction can increase error if the subsequent future patterns differ. We also observed cases where the future patterns suggested by retrieved episodes point in different directions, causing their combined result to deviate from the actual future. These limitations are also consistent with the design motivation for OREM's clipping mechanism, which bounds the magnitude of excessive corrections. We will discuss these limitations in the revised manuscript.
Thanks for the detailed response. The added baselines and multi seed results address most of my concerns, and the window level degradation analysis is helpful too. I’m more positive about the paper now. I still have a couple of questions.
-
I agree that OREM makes fewer harmful corrections in these comparisons. But for ETTm1/PatchTST, the degradation rate times the conditional MSE increase is about 0.0083 for OREM and 0.0041 for TAFAS. This only counts the losses, not the net improvement. Could you comment on this trade off and keep it clear in the reliability claims?
-
You mention failures where similar inputs lead to different futures. I was also asking about cases where there are no good neighbours in the memory. Are those queries more likely to get worse after correction?
Response to Reviewer hmXe (1/2): W1 / Q1
W1 / Q1
We appreciate the reviewer's careful evaluation and valuable suggestions, which motivated us to conduct additional experiments and investigate OREM's robustness and limitations more thoroughly. To address the reviewer's concern regarding the absence of comparisons with direct competitors, we additionally evaluated OneNet, FSNet, RAFT, and GTR across six datasets and four forecast horizons. For GTR, we considered OLS and TQNet backbones.
Table R1. Comparison with online adaptation and retrieval-based forecasters (MSE ↓).
Each value is the mean over H∈{96,192,336,720}. All methods use the same data splits and evaluation windows.
The results suggest that OREM generally achieves favorable forecasting performance relative to the compared methods.
Unlike OREM, which can be attached to a general-purpose forecasting backbone, OneNet, FSNet, and RAFT are standalone time-series forecasting models. We therefore conducted an additional experiment to examine whether applying OREM to these three methods would yield further performance improvements.
Table R2. Performance before and after applying OREM (MSE ↓).
L=H=96.
The additional experiments confirmed effective performance improvements from applying OREM to all three models. In particular, further improvements were observed for OneNet and FSNet while retaining their existing online adaptation mechanisms, and the same trend was found for the retrieval-based forecaster RAFT.
PFRP, unlike the methods above, is designed for univariate forecasting. For a fair comparison, we constructed univariate backbones by independently training each of the seven variables in the ETT datasets and provided OREM and PFRP with identical initial checkpoints.
Table R3. Comparison with PFRP on all seven ETT variables (MSE ↓).
In the comparison with PFRP, OREM also achieved the lowest MSE on every ETT dataset for both TQNet and PatchTST, without retraining the forecaster or training an additional retrieval-and-fusion module. We appreciate the reviewer's constructive suggestion, which enabled us to better understand OREM's advantages through a broader set of comparisons.