- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper looks at frozen time-series foundation models (TimesFM 2.5 and TimeMoE-50M) used with a very short target history (12 steps). It asks whether prepending windows from an external archive (ETTh2, Exchange, METR-LA, 900 windows) can help. Instead of retrieving by similarity, the authors label source-query pairs with “forecasting utility”, meaning the drop in MAE when the source is prepended, using 256 labeled fitting queries. They fit five scorers on these labels (two query-independent global scores, a reference-based kNN average, and linear/MLP regressors) and pick one on a chronological selection set (Auto-TSDM). On six targets, Auto-TSDM beats Target-Only-12 in 11 of 12 settings and beats similarity, DTW, MMD, TS2Vec and a naive RAF baseline in most settings. Target-Only-48 and LoRA remain better in 9/12 and 7/12 settings, which the paper reports openly.
Critical strengths
Strength 1. The evaluation protocol is careful. Fit, selection and test splits are chronological, test futures are never used for selection, and the selection procedure is fixed in advance. It also includes Target-48, LoRA, and direct predictors trained on the same labels. The paper also reports where the method loses (Target-48, LoRA, TimeMoE-Weather at +23.8% over Target-12).
Strength 2. The basic idea of scoring context by its measured effect on a fixed forecaster, and not by how similar it looks, is sensible and easy to deploy with black-box forecasters. On their own terms, the numbers in Table 1 support “utility labels beat unsupervised similarity at picking prefixes”.
Critical weaknesses
Weakness 1. My main concern is that the paper’s own controls point to a different mechanism than the story in the introduction. The motivation is that “temporal shapes such as peaks, recoveries, oscillations” recur across domains. But GP/GR, which use the same three sources for every query, are selected in 41/60 runs. Shuffling the time order inside source windows improves the MLP in 9/12 conditions (Table 9). Exchange-rate windows alone are the best archive for Solar on TimesFM (Table 3). Retrieving the target channel’s own past by cosine similarity is worse than Target-12 in most settings. To me this does not look like retrieval of useful temporal analogies. It looks like finding a fixed prefix that shifts the backbone’s internal normalization/scale estimate for a 12-point window (about half of the Solar test histories are constant, per C.4). Then “retrieval” is really a search over a discrete input prompt. The fair baselines for that are a tuned constant or mean prefix, a few synthetic prefixes, or a learned soft prefix. ARM’s “no source” and “constant slots” variants (Table 9) already match or beat Auto on Solar, PEMS04 and Weather for both backbones, which supports this reading. The numbers are fine, but the cross-domain temporal context framing is not well supported.
Weakness 2. The comparison against similarity baselines is not matched on supervision. Auto-TSDM uses 384 labeled target queries plus about 39.7k backbone calls to build utility labels. The similarity methods use no target labels at all. When supervision is matched (Table 2), a ridge regression or DLinear on the same 12-point histories is close or better on several targets (for example PEMS04, where Ridge gets 0.252 against Auto’s 0.270 on TimesFM, and Metro-NYC, where XGBoost gets 0.781 against Auto’s 0.991). The 12-step setting itself is also very narrow. Target-48 on Metro-NYC is 0.31 against 0.99 for Auto, so in practice a user would get much more from a slightly longer history. I would like a clearer argument for when someone really has only 12 target observations but also has 384 labeled future windows.
Minor issue : some Auto-TSDM standard deviations are large compared with the gains (PEMS-BAY 0.706±0.048 against 0.715 for Target-12), so a few of the “11 of 12” counts rest on small differences. Sections 5.3 and 5.4 are dense with hedged counts and are hard to read.
Questions for authors
-
Can you add a prefix baseline that does not use the archive, for example the best fixed constant, a mean or linear-trend prefix, or a prefix tuned by search on the fitting set? If GP/GR do no better than this, what is retrieval adding?
-
Given that time-shuffled sources do as well or better, how much of the gain is explained by the change in the backbone’s instance normalization statistics? Could you report results where the prefix is rescaled to match the query’s mean and std?
-
How does the gap to Target-Only-12 change with target history 24 or 36? I suspect it disappears quickly. If so, the paper should say that directly.