- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes SAAF, a training-free post-hoc module that lets a frozen TSFM use information beyond a short input window. A HiPPO state accumulates the whole past and feeds an online linear forecaster (CAFM), a second online linear map corrects the TSFM output (FTM), and the two are combined with an error-based weight; all updates are closed-form Bayesian linear regression at inference. Experiments on ETT and 12 Monash datasets with TimesFM (other backbones in the appendix) show clear MAE gains at short context lengths but not at long ones.
Critical strengths
-
The method needs no training, no backpropagation, and no access to TSFM internals, so it attaches to any off-the-shelf TSFM at negligible cost, and it still gives clear MAE gains at short context lengths across several backbones. This simplicity is the main reason I'd consider the paper.
-
The error-based weighting makes the adaptation respond to the current situation online, relying on the corrected TSFM output early and shifting to the accumulated state once it has warmed up, rather than applying a fixed correction.
Critical weaknesses
-
I'm not convinced the method actually improves the TSFM. In Fig. 3, SAAF is worse than No adapt (and than its own FTM component) for on ETT-h1, h2, and m2, and the claim in Sec. 5.5 that it also wins at large on m2 doesn't match the figure. Since the paper's own motivation is that TSFMs can't use information beyond the window, which holds at any , I'd expect the gains to carry over to longer windows as well; if they only appear at short , it looks like SAAF is compensating for short-window degradation (which FTM alone does) rather than exploiting beyond-window context. The gate doesn't help here either, since with CAFM keeps roughly 10-20% weight even when clearly worse.
-
The evaluation is narrower than it should be for a plug-in method whose core claim is that it helps any TSFM: one backbone and two baselines in the main text, and only ETT plus 12 Monash datasets rather than a broad benchmark. The Monash subset was selected for "sufficient records and timesteps," which is exactly the long- regime where CAFM can warm up (Sec. 5.5, Appendix A), so the dataset choice favors the method by construction and needs justification.
-
The method has many hyperparameters (, , , , ), all set once on ETTh1, which is also a test dataset (Sec. 5.3). The paper's own Fig. 14 shows the setting doesn't transfer to Monash, and in an online setting there is no validation data to tune on, so the reported numbers are open to cherry picking.
-
Fonts in Tables and Figs are far too small; most per-dataset numbers can't be read without heavy zooming.
-
Section 4 is a sequence of notation and equations with little explanation of what each component does, and most of the notation isn't needed to understand the method. It's hard to follow.
Questions for authors
-
(Clarification) Appendix E.2 states that the Monash datasets were not used to pre-train Chronos or TimesFM. Could you verify this per dataset? As far as I know, several of the selected datasets (e.g., London Smart Meters, KDD Cup 2018, Pedestrian Counts, Wind Farms) are in Chronos's training corpus, and TimesFM 2.0 was trained with LOTSA, which includes Monash. If there is overlap, please report results restricted to the non-overlapping datasets.
-
(Suggestion) A compact summary across all backbones and baselines (e.g., mean MAE and average rank per backbone) in the main text would make the generality claim much easier to assess.
For other points, please refer to the weaknesses.
3 comments
Weakness 4: Fonts in Tables and Figs are far too small; most per-dataset numbers can't be read without heavy zooming.
We apologize for the inconvenience. We adjusted figures and tables in the revised paper.
Weakness 5: Section 4 is a sequence of notation and equations with little explanation of what each component does, and most of the notation isn't needed to understand the method. It's hard to follow.
We apologize for the inconvenience. We will update the explanation in Sec. 4 to clarify what each component does during the rebuttal period.
Question 1: (Clarification) Appendix E.2 states that the Monash datasets were not used to pre-train Chronos or TimesFM. Could you verify this per dataset? As far as I know, several of the selected datasets (e.g., London Smart Meters, KDD Cup 2018, Pedestrian Counts, Wind Farms) are in Chronos's training corpus, and TimesFM 2.0 was trained with LOTSA, which includes Monash. If there is overlap, please report results restricted to the non-overlapping datasets.
Thank you for pointing it out. We acknowledge that we used TimesFM 2.0 in our experiments but mistakenly referred to the paper describing TimesFM 1.0. We also confirm that, as the reviewer correctly noted, some of the Monash datasets used in our evaluation were included in Chronos's pre-training corpus. We apologize for these inaccuracies and corrected the description in Appendix E.2. We also explicitly acknowledged the potential overlap between the TSFMs' pre-training datasets and our evaluation datasets in Sec. 5.1.
The main purpose of our experiments is not to compare the forecasting accuracy of different TSFMs, but to evaluate adaptation methods applied to the same backbone TSFM. Even on Monash, which is also used to pre-train TimesFM, largely affects the performance, as we observed in Tab. 2. Therefore, we believe that potential overlap with pre-training data does not necessarily invalidate the relative comparisons between adaptation methods, as they are evaluated using the same backbone and observations.
To further address this concern, we conducted additional experiments on GIFT-Eval. We added the results to Appendix F.10. We observed similar trends to those on ETT and Monash, with SAAF providing particularly pronounced improvements when the input window length is short.
Question 2: (Suggestion) A compact summary across all backbones and baselines (e.g., mean MAE and average rank per backbone) in the main text would make the generality claim much easier to assess.
Thank you for your suggestion. We will add a compact summary to the main paper during the rebuttal period.
Thanks for the response and the revision. I read the updated paper, but my main concerns remain.
The degradation at long context lengths is still there, and it also shows up on Monash and the new GIFT-Eval results, so short series alone can’t explain it. The response explains this by a delayed adjustment of the weights after error spikes in FTM, but this is only a hypothesis with no supporting experiment, and the paper still gives a different reason (CAFM not being warmed up). My point about the gate keeping weight on CAFM even when it is clearly worse was not addressed. It is still unclear why a method motivated by beyond-window information hurts once the window gets longer.
The Monash selection is still not justified, and the new analysis on total timesteps actually confirms that the method needs long series to win. The GIFT-Eval experiment covers only a few configurations from traffic datasets with one backbone, so it is too small to count as a broad benchmark evaluation, and SAAF doesn’t beat No adapt on the shorter series there either.
On hyperparameters, my concern was not per-dataset tuning. The configuration was chosen by looking at ETTh1, which is one of the test sets, and that is not a valid way to select it. It should be chosen on data that is not used for evaluation. The response also agrees that alpha and beta behave differently across benchmarks.
For these reasons I’m keeping my score.
Dear reviewer
kEVX,We appreciate your constructive comments. We updated the manuscript based on your comments (highlighted in blue).
Thank you for pointing it out. We revised the statement about ETT-m2 in Sec. 5.5 accordingly.
When the input window length L is large, the TSFM backbone generally produces more accurate forecasts, thereby improving the performance of FTM. However, we observed occasional spikes in FTM's forecasting error. During these periods, DWM assigns a larger weight to CAFM. Because DWM estimates each module's reliability using an exponential moving average of past errors, the increased weight assigned to CAFM can persist. This delayed adjustment can lead to higher forecasting errors for SAAF, particularly when CAFM is less accurate than FTM. In our experiments, we used the same Bayesian linear regression hyperparameters (α and β) for both CAFM and FTM. We hypothesize that tuning these hyperparameters separately for each module could improve the stability of online regression updates, potentially mitigating the resulting performance degradation.
We will report a compact summary that includes the other TSFMs in the main paper (Question 2).
Although SAAF can increase forecasting errors when T is small, the performance degradation by T is limited owing to DWM. To investigate the effect of the total number of time steps, T, on forecasting performance, we conducted an additional analysis using the Monash datasets. Specifically, we varied T and computed the accumulated mean absolute error (MAE) over the forecasting period truncated at each T. We added the results to Appendix F.6.
For (a) London Smart Meters and (b) Australian Electricity Demand, SAAF exhibits increased forecasting errors when T is small. The error peaks of SAAF and CAFM occur at approximately the same T values, suggesting that this behavior is attributable to CAFM. Nevertheless, SAAF achieves the lowest MAE among the baselines for T≥1000 across both datasets, demonstrating that its performance advantage is maintained even as T decreases. For (c) Wind Farms, SAAF exhibits higher forecasting errors than the baselines, mainly due to the relatively poor performance of CAFM. However, the additional degradation associated with small T is limited.
We would like to clarify that our results were not obtained through dataset-specific hyperparameter cherry-picking. We used the same hyperparameter configuration for the remaining ETT datasets and the Monash datasets, without any additional tuning. Therefore, the reported results were not obtained through dataset-specific hyperparameter selection.
As shown in the sensitivity analysis on the Monash datasets (Fig. 14), the effects of α and β differ notably from those observed on the ETT datasets. However, the other hyperparameters exhibit similar trends across the two benchmarks, suggesting that their effects generalize across datasets.
Additionally, we evaluated SAAF on GIFT-Eval using the same hyperparameter configuration. We confirmed its effectiveness (Appendix F.10), providing additional evidence that SAAF can generalize to other benchmarks without dataset-specific tuning.