- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper compares simple linear models (e.g., OLS, FITS, DLinear) and nonlinear deep learning models (PatchTST, TimeMixer, PatchMLP) against closed-form Ridge regression paired with an extensive preprocessing and hyperparameter tuning pipeline called SearchCast. By searching over lookback length, trailing-window local normalization, regularization, and noise augmentation across grouped horizons and series, SearchCast outperforms the compared linear and nonlinear baselines on 7 out of 8 standard benchmarks
Critical strengths
- This paper builds directly on prior work showing that linear models can achieve strong forecasting performance and that many recent linear variants are functionally equivalent to linear regression over augmented feature sets. I really appreciate the approach that rather than proposing yet another linear architecture, the authors usefully focus on the preprocessing side.
- Although I have some concerns regarding the evaluation setup, the empirical results generally support the claim that SearchCast improves over standard linear and nonlinear baselines on the considered benchmarks at a low computational cost.
Critical weaknesses
- There is a noticeable discrepancy between the framing in the Abstract/Introduction and the actual evaluation. The opening claims suggest that tuning preprocessing on a linear model can close the accuracy gap with very high-capacity architectures, explicitly citing general-purpose foundation models (e.g., first five lines of the Abstract and Section 2). However, the evaluation (a) does not include any foundation models or truly high-capacity models, and (b) relies on small/standard LTSF benchmarks where 4 of the 8 datasets (ETTh1, ETTh2, ETTm1, ETTm2) come fromvery similar sources and on the largest, highest-dimensional benchmark (Traffic, 862 series), SearchCast falls behind PatchTST.
- None of the baseline methods are given access to the key preprocessing improvements introduced for SearchCast. While running a full per-group search for deep models is computationally expensive at least the trailing-window local normalization (r<1) should have been evaluated on the baselines, especially given the paper's central finding that "full-window normalization is essentially never selected."
- The text implies that time- and frequency-domain noise augmentation is a meaningful contributor to performance, yet Figure 8(a) shows that Freq, Time, and None are each selected roughly one-third of the time across datasets, which resembles uniform random sampling. Furthermore, also from a theoretical standpoint, adding zero-mean noise to inputs in closed-form Ridge regression acts similarly to regularization, which should be largely redundant given the grid search over
Questions for authors
Beyond the weaknesses above, I would like to ask the authors:
- Table 6 gives standard deviations only for dataset averages, and only for SearchCast and the nonlinear baselines. Could you also provide standard deviations in Table 1?
- Can you provide an ablation table isolating the marginal MSE gain of each preprocessing step?
4 comments
Response to Reviewer xqY4 [2/2]
Q1. Can you report standard deviations in Table 1?
The revision moves these into Table 1, which now shows mean ± standard deviation over three runs at every horizon, for SearchCast and all rerun baselines. OLS is deterministic, and FITS and DLinear are quoted from Toner & Darlow (2024), so they, alongside TimesFM 3.0, remain single numbers.
Q2. Ablation that isolates each preprocessing step
Yes. Starting from OLS (lookback 720, full-window normalization), each column adds one step; all channels share one group until the last column (3 seeds):
| Dataset | OLS (L = 720) | + α search | + lookback search | + trailing-window norm | + augmentation | + series grouping |
|---|---|---|---|---|---|---|
| ETTh1 | 0.425 | 0.414 | 0.406 | 0.405 | 0.404 | 0.404 |
| ETTh2 | 0.334 | 0.331 | 0.332 | 0.333 | 0.334 | 0.334 |
| ETTm1 | 0.356 | 0.355 | 0.349 | 0.349 | 0.347 | 0.347 |
| ETTm2 | 0.249 | 0.248 | 0.237 | 0.238 | 0.239 | 0.239 |
| Exchange | 0.394 | 0.393 | 0.347 | 0.342 | 0.342 | 0.347 |
| Weather | 0.239 | 0.239 | 0.237 | 0.237 | 0.237 | 0.215 |
| Electricity | 0.162 | 0.162 | 0.160 | 0.160 | 0.160 | 0.160 |
| Traffic | 0.410 | 0.410 | 0.406 | 0.406 | 0.406 | 0.407 |
Removing one step at a time from the full model:
| Dataset | Full | − lookback search | − trailing-window norm | − augmentation |
|---|---|---|---|---|
| ETTh1 | 0.404 | 0.410 | 0.406 | 0.405 |
| ETTh2 | 0.334 | 0.336 | 0.332 | 0.333 |
| ETTm1 | 0.347 | 0.354 | 0.347 | 0.347 |
| ETTm2 | 0.239 | 0.249 | 0.237 | 0.238 |
| Exchange | 0.347 | 0.428 | 0.358 | 0.346 |
| Weather | 0.215 | 0.215 | 0.215 | 0.215 |
| Electricity | 0.160 | 0.162 | 0.160 | 0.160 |
| Traffic | 0.407 | 0.409 | 0.407 | 0.407 |
- Lookback search carries most of the gain. Adding it lowers MSE on 7 of 8 datasets, by up to 0.046 (Exchange), and removing it from the full model costs 0.002–0.081 on 7 of 8.
- Per-channel models carry Weather (0.237 → 0.215), whose 21 channels measure different physical quantities.
- α search matters on ETTh1 (−0.011), and trailing-window normalization on Exchange (−0.005 when added, +0.011 when removed).
- Augmentation stays within 0.0014 on the eight benchmarks and lowers MSE by 0.003 on PEMS08 (W3).
Thank you for your response. Overall I mantain my assessment: the idea is interesting, but several claims remain overstated or insufficiently supported by the evaluation. In particular:
-
Contribution (2) in the abstract and introduction claims that trailing-window normalization is almost universally preferred and "consistently improves accuracy". Which is why I asked to test it on the baselines. However, the ablation in Q2 shows that it makes essentially no difference in most of the datasets (sometimes slightly hurting, only helping in 1), while most gains come from lookback search. Moreover, W2 shows it does not help PatchTST either, which conflicts with the unchanged claim in the Conclusion that these preprocessing choices transfer to any model class.
-
Adding a single foundation model that you could not test on all the datasets provides limited support for the broad opening framing in the Abstract and Introduction (which remains unchanged in the PDF). Once Traffic and PEMS08 are excluded for pretraining overlap, 4 of the 7 evaluated datasets are ETT variants from the same source, and TimesFM 3.0 matches/slightly edges out SearchCast on Electricity
-
The response confirms W3, yet augmentation is still presented throughout the paper as a core preprocessing axis
These are prominent examples of flaws in the evaluation for which I cannot recommend acceptance.
Dear Reviewer,
The authors have kindly provided detailed responses to your comments. We would appreciate it if you could take a moment to review their rebuttal and share any further feedback or updates to your evaluation.
Thank you for your time and thoughtful consideration.
Response to Reviewer xqY4 [1/2]
Thank you for the close reading and constructive suggestions.
W1. Does the evaluation match the framing?
Yes, with the two additions you ask for. The framing is point forecasting on standard long-horizon benchmarks, and the abstract and introduction state it. The evaluation now includes a zero-shot foundation model (TimesFM 3.0), against which SearchCast is ahead on six of the seven datasets outside its pretraining corpus, and PEMS08, a dataset with 170 sensors and strong cross-channel dependence, where SearchCast is ahead of all five deep forecasters (General Q2). The revision states that the four ETT datasets share one source. On Traffic, SearchCast is within 0.010 of the best deep forecaster (0.407 vs. 0.397) at a small fraction of the training cost.
W2. Does trailing-window normalization help the deep baselines?
No. We changed PatchTST to compute its normalization mean over only the last 30% or 10% of the input window, or over the window SearchCast chose per dataset, with everything else fixed:
The best of the three windows barely changes PatchTST's MSE. With α large in most cells, Ridge shrinks the forecast toward the subtracted level, so that level acts as the forecast's anchor and choosing it matters. PatchTST's output is not shrunk toward its normalization level in this way, which may be because of its non-linear nature.
W3. Augmentations
Is augmentation actually selected, or does Figure 8a just mirror uniform sampling? Where the search selects augmentation, the noise is small: the median selected σ lies between 0.003 and 0.015 per dataset. Figure 8a counted all trials; the revised figure counts the selected configurations. At these noise levels, augmentation changes average MSE by at most 0.001 on the eight benchmarks and lowers the three-seed mean on PEMS08 by 0.003 (0.187 → 0.184). Augmentation stays in the search space as an optional axis, and the revision reports the selected levels and their effect.
Isn't zero-mean input noise in closed-form Ridge redundant with L2 regularization? For additive time-domain noise, yes: in expectation it adds Nσ² to α (Bishop, 1995). Frequency-domain noise is different: it scales the forecast by roughly 1/(1+σ²), which α alone cannot express. The search keeps both as options and selects small noise levels.