- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper propose SDTA (Shifted-window Diffusion with Temporal-distance Attention), a self-supervised pretraining framework with the model and data fixed, whose encoder is fine-tuned for downstream forecasting. SDTA encodes an input window once and trains a decoder to denoise diffusion-noised target windows, each shifted a sampled distance ahead of the input, while a temporal-distance code is added only to the attention keys. After pretraining, only the encoder is transferred and fine-tuned for forecasting.
Critical strengths
1. Simple and low-cost design
The input is encoded once and the extra targets exist only during pretraining, so memory grows at most linearly in and no cost is added at deployment.
Critical weaknesses
1. Clarity and self-containedness
The paper is difficult to follow without prior familiarity with the area. Many terms are used without definition and the related work lists prior methods without explaining what they do or how SDTA differs from them. The introduction also mixes the core idea with detailed numerical results and defensive statements.
2. Motivation for pretraining on the target dataset
In forecasting, labels are simply future values of the same series, so supervised training already uses all available data. Since pretraining and fine-tuning are performed on the same dataset, it is unclear why a separate pretraining stage is needed. Moreover, the proposed pretext, which reconstructs windows shifted ahead of the input, appears close to forecasting itself. The paper does not explain why this would yield better representations than training a forecaster directly, and no supervised forecasting model is included in the comparison.
3. Fair comparison
The protocol equalizes epochs rather than compute, and costs about three times as much per epoch as . It is therefore unclear whether the gain from increasing comes from the additional targets themselves or simply from the additional pretraining compute.
4. Magnitude of the improvement
Increasing from 1 to 4 reduces the average MSE by about 4%, and the difference from the strongest baseline, TimeMAE-CI, is not statistically significant. Given that all experiments use a very small encoder (, about 17k parameters), it is unclear whether the scaling narrative and the reported gains hold for models of practical size.
Questions for authors
- Why is a separate pretraining stage needed when pretraining and fine-tuning use the same dataset? How does SDTA compare with supervised forecasting models trained directly on the task?
- How does SDTA with compare with trained under the same total compute?
- Since the target windows overlap with the input for , how much of the pretext amounts to reconstructing content already present in the input?
- Do the gains from increasing persist with larger encoders?
5 comments
Response to Reviewer jf9A (Part 2/3)
W2 / Q1, continued. On "the pretext looks like forecasting": the targets exist only inside the pretext, at shifts unrelated to the downstream prediction lengths, and feed a decoder that is discarded before fine-tuning, and what transfers is the encoder. The comparison above is the test of whether that indirection is worth anything: at convergence it is, by 5 % with full data and 8–10 % with scarce data.
W3 / Q2: epochs rather than compute are equalized, and W = 4 vs W = 1 at the same total compute. Table 3 and Appendix I.1 answer this on ten datasets:
- W = 1 (the paper's forced s = 1 target) trained for 200 epochs, four times the budget, stays at 0.3399 vs 0.3402 at 50 epochs, while W = 4 at 50 epochs is 0.3288. Compute alone does not explain the gain.
- The gain is carried by targets at different distances. The paper's W = 1 baseline had none (its single target is always the s = 1 window). Letting that target take a random distance, at identical compute, recovers 70 % of the gap (0.3402 → 0.3322). With every target random the remaining count effect is 0.0020.
- Once a random-distance target exists, W and epochs are interchangeable: W = 2 for 100 epochs = W = 4 for 50 epochs (0.3288), and W = 1 random for 200 epochs reaches 0.3243. W is the cheaper carrier of the same supervision (0.26–0.66× the pretraining time of W = 1 for 200 epochs, Table 36), not a unique source of gain. The paper now says this. The same pattern holds under transfer with scarce data (Table 57): W = 4 ends 5.5 % below W = 1, W = 1 with four times the pretraining epochs stays at the W = 1 level (0.3889 vs 0.3848), and W = 1 with a random distance and four times the epochs reaches 0.3531.
W4 / Q4: about 4 % gain, not significant against TimeMAE-CI, and a tiny encoder (d = 32, 17k parameters). Does the gain hold at practical sizes?
- Significance against TimeMAE-CI. We never claimed it, and the picture is now worse for us, not better: under one fine-tune schedule for every method (new Table 2) TimeMAE-CI is ahead by 0.0037 on 6/10 datasets. The revision claims parity under the paper's protocol and second place under the unified one.
- The 4 %. On this benchmark it corresponds to a gain on 12/12 datasets (p = 0.0005), and the entire spread of the eight self-supervised methods is 0.04, so it is the size of the differences the field reports, and we no longer lean on it in the introduction.
- Width. Appendix I.5 (Table 50) sweeps d ∈ {32, 64, 128} at W ∈ {1, 4} on eight datasets, with TimeMAE-CI and TimeDART at d = 128. The W gain on the six low-channel datasets shrinks with width (0.0040 → 0.0025 → 0.0016) and grows on PEMS08 (0.020 → 0.030 → 0.050). Doubling the width from 32 to 64 at W = 1 (0.3340) buys as much as W = 1→4 at d = 32 (0.3339), d = 128 at W = 1 reaches 0.3309, and d = 128 at W = 4 is the best eight-dataset row (0.3223, against 0.3318 for TimeMAE-CI and 0.3326 for TimeDART at the same width). Width, epochs and W are alternative ways to add supervision. Which one pays most depends on the data, and on the high-channel PEMS08 the W gain grows with width rather than shrinking.
Q3: targets overlap the input when s < 12. How much of the pretext is reconstructing content already in the input? None of it, by construction, and the revised Section 3.2 now says why. The decoder's cross-attention mask is index-based: query patch i of a target may attend only anchor keys p ≤ i, which are raw patches a+1, …, a+i, while the patch it must reconstruct is raw patch a+s+i. Every target patch is predicted from context that ends at least s patches before it, for every s ≥ 1, so the overlap between the windows never places the answer among the visible keys. This is also why the encoder has to be causal (Section 3.2). The pretext is a prediction problem at distance s, not a reconstruction of visible content. A sweep of s_min (non-overlapping targets only) would make this quantitative. We did not run it in the discussion period and list it as open.
Response to Reviewer jf9A (Part 3/3)
Positive signals we would like to point to.
- Pretraining pays at convergence, and it transfers: 5.2 % below the same encoder trained from scratch with full data, 8–10 % below with scarce data and an encoder pretrained on another traffic network, about half of that from another domain, on every dataset–seed pair (Figure 3, Tables 57–59).
- The controlled experiments give a clear mechanism: with W, epochs and seeds fixed, moving the single target from the forced s = 1 window to a random distance changes the 10-ds MSE from 0.3399 to 0.3243, with gaps of 0.04–0.06 on each PEMS dataset. This is the part of the paper's claim that survived every control.
- All paper rows reproduce on a different machine and PyTorch version within 0.0003 (Table 39).
- The shared pretraining learning rate is the best of three for the runner-up and below the best for SDTA (Table 44), and the baselines do not improve with three times the pretraining budget (Table 43).
- At twice the longest standard horizon SDTA W = 4 keeps the best mean of seven methods (0.5640, Table 54), and at d = 128 it is the best row at that width (Table 50).
Your objection exposed a real weakness: the submission's evidence for pretraining was a 10-epoch control, and we no longer rely on it. What the paper now offers is a capacity-matched comparison of eight pretexts under two fine-tune protocols, a mechanism study of what the multi-window axis scales (and what it does not), the from-scratch and width controls that were missing, with both sides trained to convergence, and a transfer study across networks and domains. We would be grateful if you could reassess the paper on that basis.
Response to Reviewer jf9A (Part 1/3)
Thank you for a direct review. Your main objection, that a separate pretraining stage on the same dataset needs a justification the paper did not give, is one we took seriously enough to test. The submission's own control did not survive the test. A comparison with both sides trained to convergence does, and the paper is rewritten around it. We answer each point with the new evidence.
W1: clarity and self-containedness. The revision rewrites the introduction and the abstract: every term is defined at first use (anchor, target, shift s and s_max, supervision axis, memory-feasible, capacity-matched), the numbers and the defensive statements are gone, no sentence exceeds 35 words, and Section 2 gives one clause per baseline on what it reconstructs and where its target sits, with TS2Vec, CoST and AutoCon added. Section 3 now states the overlap between anchor and target for s < 12, the instance-normalization used, and the index-based cross-attention mask, and Algorithm 1 is indexed per patch with m and s_min in its parameter list.
W2 / Q1: why a separate pretraining stage on the same dataset, the pretext looks like forecasting, and no supervised forecaster is in the comparison. Short answer: a pretrained encoder ends lower than the same encoder trained directly on the forecasting task once both are trained to convergence, and the pretraining data need not be the target's. The paper compares pretexts at matched capacity, so the supervised counterpart it adds is the capacity-matched one: the same patch-based, channel-independent encoder and head, trained on the forecasting task from random initialization. Three results, five seeds (Section 6, new Figure 3, Appendix I.4 and I.8):
At the paper's budget, the control you asked for closes the gap. The paper's random-init control trained for the 10 fine-tune epochs only. Trained for 60 epochs, the total of pretraining and fine-tuning, the same encoder reaches 0.3296 against 0.3288 for pretrain-then-fine-tune on ten datasets (Table 48). On this point your reading was correct, and we no longer use the 10-epoch control as evidence.
Neither side of that comparison has converged, and at convergence pretraining is ahead. A 10-epoch fine-tune stops with its validation error still falling. On PEMS03/04/08 we trained both to convergence: pretraining plus 60 fine-tune epochs against 110 epochs from scratch, the same total, both with early stopping.
Given 260 epochs, the from-scratch encoder is still 8.0 % behind at 10 % (0.3954). At 5 % the gap narrows to 1.7 %.
The pretraining data need not be the target's. This is our answer to "supervised training already uses all available data": the pretext can read series for which the forecaster has no labels. With 10 % of the target's windows, an encoder pretrained on another traffic network ends 10.3 % below training from scratch, and one pretrained on electricity or weather 5.5 % and 4.5 % below (0.3833 and 0.3876), on every pair.
The paper states the limits. The convergence comparison covers the three PEMS benchmarks. The gain grows with the horizon and is absent at prediction length 12. With 10 % of the windows of the four ETT datasets transfer does not pay for any of the three pretexts (SDTA 0.4657 against 0.4500 from scratch). Among pretexts the best one depends on the data: TimeMAE-CI on PEMS (0.3571 at 10 %) and TimeDART on ETT (0.4520), with SDTA second on both.