acceptodds
Under review as a conference paper at ICLR 2027

Shifted-window Diffusion with Temporal-distance Attention for Self-Supervised Time-Series Forecasting

Abstract

Scaling drives modern machine learning, but its usual currency is parameters, data, and GPU memory. Holding the model and data fixed, we instead scale the supervision, namely the number of prediction targets per input. Existing self-supervised pretexts for time series reconstruct targets within a single window or span at most two, so none of them can scale this axis. We propose SDTA (Shifted-window Diffusion with Temporal-distance Attention), a self-supervised pretraining framework whose encoder is fine-tuned for downstream forecasting. SDTA encodes the input window once and denoises diffusion-noised target windows, each shifted a sampled distance into the future, so that one encoding must support prediction at many temporal distances. Temporal-distance attention injects each target's distance only into the attention keys, steering where the decoder attends rather than what it copies. Because the input is encoded once and the targets exist only in pretraining, raising multiplies supervision at linear activation memory, fits one consumer GPU, and adds no deployment cost. Experiments on twelve standard forecasting benchmarks spanning energy, weather, finance, and traffic compare eight self-supervised methods under identical encoder capacity, one fixed configuration, no per-dataset tuning, and five-seed averaging. We further scale each baseline's own supervision axis: most possess no memory-feasible counterpart, and the two that do gain nothing from scaling it, whereas SDTA improves consistently along this axis, with the gain widening beyond the standard horizons. SDTA also achieves the lowest average MSE and MAE among the eight methods, and ranks top-two on ten of the twelve benchmarks by MSE and nine by MAE.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, self-supervised learning, pre-training, representation learning, diffusion models, scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jf9AOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose SDTA (Shifted-window Diffusion with Temporal-distance Attention), a self-supervised pretraining framework with the model and data fixed, whose encoder is fine-tuned for downstream forecasting. SDTA encodes an input window once and trains a decoder to denoise WW diffusion-noised target windows, each shifted a sampled distance ahead of the input, while a temporal-distance code is added only to the attention keys. After pretraining, only the encoder is transferred and fine-tuned for forecasting.

Critical strengths

1. Simple and low-cost design

The input is encoded once and the extra targets exist only during pretraining, so memory grows at most linearly in WW and no cost is added at deployment.

Critical weaknesses

1. Clarity and self-containedness

The paper is difficult to follow without prior familiarity with the area. Many terms are used without definition and the related work lists prior methods without explaining what they do or how SDTA differs from them. The introduction also mixes the core idea with detailed numerical results and defensive statements.

2. Motivation for pretraining on the target dataset

In forecasting, labels are simply future values of the same series, so supervised training already uses all available data. Since pretraining and fine-tuning are performed on the same dataset, it is unclear why a separate pretraining stage is needed. Moreover, the proposed pretext, which reconstructs windows shifted ahead of the input, appears close to forecasting itself. The paper does not explain why this would yield better representations than training a forecaster directly, and no supervised forecasting model is included in the comparison.

3. Fair comparison

The protocol equalizes epochs rather than compute, and W=4W=4 costs about three times as much per epoch as W=1W=1. It is therefore unclear whether the gain from increasing WW comes from the additional targets themselves or simply from the additional pretraining compute.

4. Magnitude of the improvement

Increasing WW from 1 to 4 reduces the average MSE by about 4%, and the difference from the strongest baseline, TimeMAE-CI, is not statistically significant. Given that all experiments use a very small encoder (d=32d=32, about 17k parameters), it is unclear whether the scaling narrative and the reported gains hold for models of practical size.

Questions for authors
  1. Why is a separate pretraining stage needed when pretraining and fine-tuning use the same dataset? How does SDTA compare with supervised forecasting models trained directly on the task?
  2. How does SDTA with W=4W=4 compare with W=1W=1 trained under the same total compute?
  3. Since the target windows overlap with the input for s<12s<12, how much of the pretext amounts to reconstructing content already present in the input?
  4. Do the gains from increasing WW persist with larger encoders?
5 comments
AuthorsOpenReview ↗

Response to Reviewer jf9A (Part 1/3)

Thank you for a direct review. Your main objection, that a separate pretraining stage on the same dataset needs a justification the paper did not give, is one we took seriously enough to test. The submission's own control did not survive the test. A comparison with both sides trained to convergence does, and the paper is rewritten around it. We answer each point with the new evidence.

W1: clarity and self-containedness. The revision rewrites the introduction and the abstract: every term is defined at first use (anchor, target, shift s and s_max, supervision axis, memory-feasible, capacity-matched), the numbers and the defensive statements are gone, no sentence exceeds 35 words, and Section 2 gives one clause per baseline on what it reconstructs and where its target sits, with TS2Vec, CoST and AutoCon added. Section 3 now states the overlap between anchor and target for s < 12, the instance-normalization used, and the index-based cross-attention mask, and Algorithm 1 is indexed per patch with m and s_min in its parameter list.

W2 / Q1: why a separate pretraining stage on the same dataset, the pretext looks like forecasting, and no supervised forecaster is in the comparison. Short answer: a pretrained encoder ends lower than the same encoder trained directly on the forecasting task once both are trained to convergence, and the pretraining data need not be the target's. The paper compares pretexts at matched capacity, so the supervised counterpart it adds is the capacity-matched one: the same patch-based, channel-independent encoder and head, trained on the forecasting task from random initialization. Three results, five seeds (Section 6, new Figure 3, Appendix I.4 and I.8):

At the paper's budget, the control you asked for closes the gap. The paper's random-init control trained for the 10 fine-tune epochs only. Trained for 60 epochs, the total of pretraining and fine-tuning, the same encoder reaches 0.3296 against 0.3288 for pretrain-then-fine-tune on ten datasets (Table 48). On this point your reading was correct, and we no longer use the 10-epoch control as evidence.

Neither side of that comparison has converged, and at convergence pretraining is ahead. A 10-epoch fine-tune stops with its validation error still falling. On PEMS03/04/08 we trained both to convergence: pretraining plus 60 fine-tune epochs against 110 epochs from scratch, the same total, both with early stopping.

PEMS03/04/08, 3-ds MSE (Tables 57–58)from scratch, 110 epochspretrained, fine-tuned 60 epochs
full training split, pretrained on the target0.31120.2950−5.2 %, 15/15 dataset–seed pairs
25 % of the windows, pretrained on another traffic network0.35250.3195−9.3 %, 15/15
10 %0.40580.3638−10.3 %, 15/15
5 %0.48700.4489−7.8 %, 15/15

Given 260 epochs, the from-scratch encoder is still 8.0 % behind at 10 % (0.3954). At 5 % the gap narrows to 1.7 %.

The pretraining data need not be the target's. This is our answer to "supervised training already uses all available data": the pretext can read series for which the forecaster has no labels. With 10 % of the target's windows, an encoder pretrained on another traffic network ends 10.3 % below training from scratch, and one pretrained on electricity or weather 5.5 % and 4.5 % below (0.3833 and 0.3876), on every pair.

The paper states the limits. The convergence comparison covers the three PEMS benchmarks. The gain grows with the horizon and is absent at prediction length 12. With 10 % of the windows of the four ETT datasets transfer does not pay for any of the three pretexts (SDTA 0.4657 against 0.4500 from scratch). Among pretexts the best one depends on the data: TimeMAE-CI on PEMS (0.3571 at 10 %) and TimeDART on ETT (0.4520), with SDTA second on both.

AuthorsOpenReview ↗

Response to Reviewer jf9A (Part 2/3)

W2 / Q1, continued. On "the pretext looks like forecasting": the targets exist only inside the pretext, at shifts unrelated to the downstream prediction lengths, and feed a decoder that is discarded before fine-tuning, and what transfers is the encoder. The comparison above is the test of whether that indirection is worth anything: at convergence it is, by 5 % with full data and 8–10 % with scarce data.

W3 / Q2: epochs rather than compute are equalized, and W = 4 vs W = 1 at the same total compute. Table 3 and Appendix I.1 answer this on ten datasets:

  • W = 1 (the paper's forced s = 1 target) trained for 200 epochs, four times the budget, stays at 0.3399 vs 0.3402 at 50 epochs, while W = 4 at 50 epochs is 0.3288. Compute alone does not explain the gain.
  • The gain is carried by targets at different distances. The paper's W = 1 baseline had none (its single target is always the s = 1 window). Letting that target take a random distance, at identical compute, recovers 70 % of the gap (0.3402 → 0.3322). With every target random the remaining count effect is 0.0020.
  • Once a random-distance target exists, W and epochs are interchangeable: W = 2 for 100 epochs = W = 4 for 50 epochs (0.3288), and W = 1 random for 200 epochs reaches 0.3243. W is the cheaper carrier of the same supervision (0.26–0.66× the pretraining time of W = 1 for 200 epochs, Table 36), not a unique source of gain. The paper now says this. The same pattern holds under transfer with scarce data (Table 57): W = 4 ends 5.5 % below W = 1, W = 1 with four times the pretraining epochs stays at the W = 1 level (0.3889 vs 0.3848), and W = 1 with a random distance and four times the epochs reaches 0.3531.

W4 / Q4: about 4 % gain, not significant against TimeMAE-CI, and a tiny encoder (d = 32, 17k parameters). Does the gain hold at practical sizes?

  • Significance against TimeMAE-CI. We never claimed it, and the picture is now worse for us, not better: under one fine-tune schedule for every method (new Table 2) TimeMAE-CI is ahead by 0.0037 on 6/10 datasets. The revision claims parity under the paper's protocol and second place under the unified one.
  • The 4 %. On this benchmark it corresponds to a gain on 12/12 datasets (p = 0.0005), and the entire spread of the eight self-supervised methods is 0.04, so it is the size of the differences the field reports, and we no longer lean on it in the introduction.
  • Width. Appendix I.5 (Table 50) sweeps d ∈ {32, 64, 128} at W ∈ {1, 4} on eight datasets, with TimeMAE-CI and TimeDART at d = 128. The W gain on the six low-channel datasets shrinks with width (0.0040 → 0.0025 → 0.0016) and grows on PEMS08 (0.020 → 0.030 → 0.050). Doubling the width from 32 to 64 at W = 1 (0.3340) buys as much as W = 1→4 at d = 32 (0.3339), d = 128 at W = 1 reaches 0.3309, and d = 128 at W = 4 is the best eight-dataset row (0.3223, against 0.3318 for TimeMAE-CI and 0.3326 for TimeDART at the same width). Width, epochs and W are alternative ways to add supervision. Which one pays most depends on the data, and on the high-channel PEMS08 the W gain grows with width rather than shrinking.

Q3: targets overlap the input when s < 12. How much of the pretext is reconstructing content already in the input? None of it, by construction, and the revised Section 3.2 now says why. The decoder's cross-attention mask is index-based: query patch i of a target may attend only anchor keys p ≤ i, which are raw patches a+1, …, a+i, while the patch it must reconstruct is raw patch a+s+i. Every target patch is predicted from context that ends at least s patches before it, for every s ≥ 1, so the overlap between the windows never places the answer among the visible keys. This is also why the encoder has to be causal (Section 3.2). The pretext is a prediction problem at distance s, not a reconstruction of visible content. A sweep of s_min (non-overlapping targets only) would make this quantitative. We did not run it in the discussion period and list it as open.

AuthorsOpenReview ↗

Response to Reviewer jf9A (Part 3/3)

Positive signals we would like to point to.

  • Pretraining pays at convergence, and it transfers: 5.2 % below the same encoder trained from scratch with full data, 8–10 % below with scarce data and an encoder pretrained on another traffic network, about half of that from another domain, on every dataset–seed pair (Figure 3, Tables 57–59).
  • The controlled experiments give a clear mechanism: with W, epochs and seeds fixed, moving the single target from the forced s = 1 window to a random distance changes the 10-ds MSE from 0.3399 to 0.3243, with gaps of 0.04–0.06 on each PEMS dataset. This is the part of the paper's claim that survived every control.
  • All paper rows reproduce on a different machine and PyTorch version within 0.0003 (Table 39).
  • The shared pretraining learning rate is the best of three for the runner-up and below the best for SDTA (Table 44), and the baselines do not improve with three times the pretraining budget (Table 43).
  • At twice the longest standard horizon SDTA W = 4 keeps the best mean of seven methods (0.5640, Table 54), and at d = 128 it is the best row at that width (Table 50).

Your objection exposed a real weakness: the submission's evidence for pretraining was a 10-epoch control, and we no longer rely on it. What the paper now offers is a capacity-matched comparison of eight pretexts under two fine-tune protocols, a mechanism study of what the multi-window axis scales (and what it does not), the from-scratch and width controls that were missing, with both sides trained to convergence, and a transfer study across networks and domains. We would be grateful if you could reassess the paper on that basis.

Reviewer_pu7NOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SDTA, a self-supervised pretraining framework for time-series forecasting. It extends diffusion reconstruction from one temporal window to multiple shifted future target windows with different offsets: the anchor input is encoded once, and this representation is reused for all targets, which are reconstructed via a diffusion decoder. A continuous temporal-distance code is added to cross-attention keys for distance-aware modeling. After pretraining, only patch embedding and the causal encoder are transferred for forecasting; the diffusion decoder and distance encoder are discarded. SDTA is compared with eight self-supervised baselines on twelve benchmarks under a capacity-matched setup without dataset-specific tuning, achieving the best average MSE and MAE. More target windows W improve accuracy, adding cost only in pretraining with no inference overhead, a contrast to baselines that fail to benefit from larger supervision budgets.

Critical strengths
  1. The method is clearly motivated and integrates diffusion modeling with multi-window forecasting within a coherent framework to capture multi-scale temporal patterns.
  2. Experiments cover 12 datasets and multiple self-supervised baselines, with ablation studies verifying the effectiveness of the core designs.
  3. The diffusion decoder and temporal-distance modules are discarded in downstream inference, substantially lowering deployment overhead.
Critical weaknesses
  1. The proposed framework combines diffusion-based prediction, multi-window forecasting, and temporal-distance-aware attention, but its distinction from existing predictive and diffusion-based pretraining methods remains insufficiently established. Since ablations suggest that diffusion contributes most of the performance gains, further evidence is needed to isolate the unique contribution of the multi-window formulation.

  2. Performance improves with more future target windows, but this also increases training time and memory usage. Compute-matched experiments are needed to disentangle the benefits of multi-window learning from those of additional computation. Comparisons with well-tuned baselines would also strengthen the fairness of the evaluation.

  3. Its performance gain is modest relative to the overall improvement. Further analysis is needed to establish whether this benefit is consistent and statistically significant across different experimental settings.

  4. The evaluation focuses primarily on in-dataset forecasting, with limited evidence of robustness under distribution shifts. The long-horizon evaluation is also limited to a single extended horizon and a reduced subset of datasets, leaving the method's generalization to more challenging forecasting settings insufficiently established.

Questions for authors
  1. The ablation results suggest that diffusion accounts for most of the performance gains. Can the authors further clarify the unique contribution of the multi-window formulation beyond standard diffusion-based prediction, and provide evidence that its benefits cannot be attributed merely to access to more future targets during training?
  2. Can the authors provide compute-matched comparisons to disentangle the benefits introduced by the multi-window design from those originating from extra computational overhead? In particular, how does SDTA compare with a single-window variant trained under an identical computational budget?
  3. Given the relatively modest improvement from temporal-distance attention, can the authors clarify under which settings this component is most beneficial, and verify whether its gains are consistent and statistically significant across datasets and forecasting horizons?
  4. How well does SDTA generalize under distribution shifts and across substantially longer forecasting horizons? The current long-horizon evaluation is limited to a single horizon on a reduced dataset subset. Can the authors clarify the scope of the current evaluation and provide additional supporting evidence where feasible?
4 comments
AuthorsOpenReview ↗

Response to Reviewer pu7N (Part 1/2)

Thank you for the detailed review and for the fairness checks you asked for. They led to the two most informative tables of the revision. The general response summarizes the new experiments, and here we answer each point.

W1 / Q1: the distinction from predictive and diffusion pretraining is unclear, diffusion seems to contribute most, so the multi-window contribution must be isolated, and shown not to be "access to more future targets". Three pieces of evidence, all on ten datasets with five seeds (Table 3, Appendix I.1):

  • More draws of the same target buy nothing. W = 4 with s_max = 1 (four independent noise draws of the same s = 1 window) equals W = 1 on all ten datasets: 0.3404 vs 0.3402. Whatever W adds is not "more corruption draws".
  • Targets at different distances are the ingredient. With the same W, epochs and seeds, moving the single target from the forced s = 1 window to a random distance changes the 10-ds MSE from 0.3399 to 0.3243 at 200 epochs (0.04–0.06 on each PEMS dataset). "More future targets" only helps when they lie at different distances, and at W = 1 the paper's baseline had none.
  • Relation to TimeDART. TimeDART is diffusion denoising of the input window itself (10-ds 0.3409 under its own schedule). SDTA W = 1 random, the closest single-window counterpart, is 0.3322 at the same budget, and the gap to SDTA W = 4 comes on top of that. The revised Section 2 states for every baseline what it reconstructs and where its target sits, and Section 6 notes that the paper's "no diffusion" arm is a clean-query shortcut (the decoder can copy the clean target), so it measures the collapse of the pretext rather than a clean diffusion-vs-other-corruption comparison. A mask-token forecasting pretext without diffusion, which would be that comparison, could not be run in the discussion period, and we say so in Section 7.
  • Multi-window against diffusion alone, under transfer. TimeDART is diffusion denoising of one window, SDTA W = 1 of one shifted window, and W = 4 adds windows. With a tenth of the target's training windows on PEMS03/04/08, every encoder pretrained on another traffic network and fine-tuned to convergence (new Figure 3, Table 57), TimeDART and SDTA W = 1 end at the same level (0.3877 and 0.3848) and W = 4 ends at 0.3638, 6.2 % and 5.5 % lower, ahead on 15 and 14 of the 15 dataset–seed pairs. The step from one window to four is the step that pays, and diffusion alone does not deliver it.

W2 / Q2: compute-matched comparison with a single-window variant, and well-tuned baselines.

  • Compute. W = 1 (forced) for 200 epochs, four times the budget, stays at 0.3399 vs 0.3402 for 50 epochs. W = 2 for 100 epochs reproduces W = 4 for 50 epochs (0.3288 = 0.3288). W = 1 with random distances for 200 epochs reaches 0.3243, i.e. with distance diversity and the number of distance-shifted targets matched, the single-window variant is as good as or better than W = 4. W is then the cheaper of the two routes (0.26–0.66× the pretraining time, Table 36), not a unique one. Section 5.2 and Section 7 now say this.
  • Baselines well tuned? Two checks. TimeDART and TimeMAE-CI pretrained for 150 instead of 50 epochs move by −0.0012 and +0.0016. A pretraining learning-rate sweep over {3e-4, 1e-3, 3e-3} (Table 44) finds the shared 1e-3 to be the best of the three for TimeMAE-CI (the alternatives cost it 0.0088 and 0.0072), within ±0.001 for TimeDART, and below the best for SDTA (which gains 0.0038 at 3e-3, better on six of nine datasets). The shared value did not favor us.
  • Fine-tune schedule. This is the check that went against the paper. Each method in Table 1 fine-tunes under its own paper's schedule (OneCycleLR for SDTA and TimeDART, step-halving for TimeMAE-CI, CPC and SimTS). Under OneCycleLR for every method (new Table 2), TimeMAE-CI improves from 0.3297 to 0.3251 and edges ahead of SDTA (0.3288) on 6/10 datasets (Wilcoxon p = 0.49), and under step-halving for every method SDTA (0.3329) also trails TimeMAE-CI (0.3297). OneCycleLR helps every baseline by 0.005–0.008. The revision keeps Table 1, adds Table 2, and claims parity under the paper's protocol and second place under the unified one (Sections 5.1, 5.6, 7).
AuthorsOpenReview ↗

Response to Reviewer pu7N (Part 2/2)

W3 / Q3: the gain from temporal-distance attention is modest. When does it help, and is it consistent and significant? We agree it is a refinement rather than a pillar, and the paper says so. The evidence we have is the paper's own ablation on twelve datasets: removing the distance code costs +0.0037 MSE, about 3.7× the cross-seed standard deviation of the default, worse on 10/12 datasets, all three dataset groups and all five paired seeds, and the cost grows to +0.0059 at the extended horizon. The discrete-token substitute costs +0.0038. The component is most useful where distances are many and the horizon is long (PEMS, traffic, ETTh1 and the extended horizon). We did not run a TDA × W interaction study in the discussion period, so we cannot say whether the gain changes with W. Section 7 lists this as open rather than claiming it.

W4 / Q4: in-dataset only, one extended horizon on a reduced subset, and distribution shift.

  • Reduced subset. The extended horizon in the submission covered eleven of twelve datasets, and only exchange is too short for a 1080-step horizon. We now say so explicitly.
  • A second, longer horizon. pred_len 1440 (192 on PEMS), nine datasets (Appendix I.7, Table 54): SDTA W = 4 keeps the best mean (0.5640). W = 1 trails by 0.0294 (Holm p = 0.047), TimeDART and CPC by ≈0.028, SimTS by 0.103, TimeMAE-CI by 0.0041 (ahead on PEMS, behind on ETT). The W gain at this horizon is 0.074 on PEMS.
  • Longer input. With input length 336 on the six low-channel datasets the W gain disappears (W = 1 0.3397 vs W = 4 0.3401), although SDTA still beats TimeMAE-CI by 0.0033 there. We report this as a limitation: the long input supplies the multi-scale context that distance-shifted targets otherwise add.
  • Distribution shift and scarce data. Two experiments (Appendix I.8). (1) Cross-dataset transfer at the paper's fine-tune, full target split (ETTh1↔ETTh2, ETTm1↔ETTm2, PEMS04→PEMS08, Table 56): the encoder ports (transfer W = 4 is 0.0035 behind in-dataset W = 4 on the five-dataset mean, and transfer W = 1 equals in-dataset W = 1), and the W gain survives the shift at about 60 % of its size (0.022 on PEMS08). (2) Transfer to scarce target data with both sides trained to convergence (new Figure 3, Tables 57–59). The target is PEMS03/04/08 with the first 5 / 10 / 25 % of its training windows. The encoder is pretrained on another traffic network and fine-tuned for 60 epochs, and the control is the same encoder trained from scratch for 110 epochs, the same total, and for 260. The pretrained encoder ends 7.8 / 10.3 / 9.3 % below training from scratch (0.3638 vs 0.4058 at 10 %) on all 45 dataset–seed pairs, and at 10 % stays 8.0 % below the 260-epoch run (0.3954). Pretrained on a different domain, electricity or weather, it keeps about half the gain (0.3833 and 0.3876). The gain grows with the horizon: none at prediction length 12, 11–15 % at 96. The limits we report: on the four ETT datasets no pretext gains from transfer (SDTA +3.5 %, TimeMAE-CI +10.0 %, TimeDART +0.4 % against their own encoders trained from scratch). Among pretexts, the best transfer depends on the data: TimeMAE-CI on PEMS (0.3571 at 10 %), TimeDART on ETT (0.4520). SDTA is second on both, at most 3.0 % behind the best, while each baseline is last on one family by 8.6–10.1 %.

Positive signals we would like to point to.

  • The paper's rows reproduce on a second machine within 0.0003 (Table 39).
  • The same-seed pair that differs only in the distance distribution (0.3399 vs 0.3243) isolates distance-shifted targets as the mechanism.
  • The shared learning rate is the best of three for the runner-up and below the best for SDTA (Table 44).
  • At twice the longest standard horizon SDTA keeps the best mean of seven methods (Table 54).
  • Pretraining pays once both sides are trained to convergence, and it transfers: on PEMS03/04/08 the pretrained encoder ends 5.2 % below the same encoder trained from scratch with full data, 8–10 % below with scarce data and an encoder from another traffic network, and about half of that with an encoder from another domain (Figure 3, Tables 57–59).
  • The unified-protocol table you asked for is, we think, an improvement to the benchmark itself: it shows that OneCycleLR is worth 0.005–0.008 to every masked or contrastive baseline, which future comparisons should adopt.

We would be grateful if you could weigh the revision, including Table 2 where the result goes against us, in your final assessment.

Thanks to the authors for carefully addressing my major concerns. Extensive additional experiments have been supplemented for the main issues I raised previously, including comparisons of computational overhead, baseline evaluations under a unified fine-tuning strategy, and cross-network few-shot transfer experiments. These additions strengthen the credibility of the experimental results and further verify the effectiveness of the proposed method in certain scenarios. Nevertheless, the experimental results do not fully demonstrate that the multi-window design itself is irreplaceable. I will keep my current positive rating.

AuthorsOpenReview ↗

Thank you Reviewer pu7N

Thank you for the careful second reading and for keeping a positive rating.

We agree with the remaining reservation and have stated it in the paper: once a random-distance target exists, W and epochs are interchangeable, so W is the cheapest carrier of distance-shifted supervision rather than an irreplaceable one.

We owe that finding to the compute-matched comparison you asked for in W2/Q2. The single-window variant at four times the epochs is what showed that distance diversity, not the window count, carries the gain, and it changed how the paper describes the axis.

What the added experiments do establish is that this axis pays only for SDTA's pretext (TimeSiam's gets worse along the same axis, Table 15) and that it keeps paying under transfer to scarce data (Table 57).

We also appreciate the fairness checks. The unified-schedule table in particular improved the benchmark itself.

Reviewer_z8oxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SDTA, a self-supervised forecasting framework that encodes an anchor window once and reconstructs multiple diffusion-corrupted, temporally shifted target windows. Temporal-distance embeddings are added to attention keys, and only the pretrained encoder and patch embedding transfer downstream. Experiments compare eight pretraining methods across twelve datasets with matched transferred encoder capacity and five seeds. SDTA reports the lowest average standard-horizon MSE and MAE, supported by window-count sweeps, ablations, extended-horizon evaluations, and memory/runtime measurements.

Critical strengths
  1. Reusing one anchor encoding for multiple shifted denoising targets is a practical design, which enriches pretraining without increasing the deployed model size. The reported forecasting results are competitive across a broad set of datasets.

  2. The evaluation is extensive, covering five seeds, component ablations, extended horizons, and memory/runtime measurements. The paper also acknowledges that its advantage over TimeMAE-CI is not statistically significant.

Critical weaknesses
  1. Algorithm 1 always forces one target to s=1s=1, so increasing WW also changes the distance distribution. The no-forced-target ablation is only evaluated at W=4W=4, leaving the claimed count-specific benefit unresolved.

  2. The claim that this scaling benefit is unique to SDTA is insufficiently controlled. Epochs, not compute, are matched, and W=4W=4 costs about three times as much as W=1W=1 on PEMS07. Compute-matched runs and repeated-noise/mask controls for single-window baselines are needed to rule out simpler explanations.

  3. The article is suspected to have extensively used AI polishing or generation, such as the use of the marker word "therefore" (totally 18 times) in sentences.

  4. The article is a bit rough, such as only one word occupies one line (lines 33, 44, 118, 439, etc.), which needs to be adjusted.

Questions for authors

Weaknesses and the following:

  1. Does W=4W=4 still outperform W=1W=1 when the target-distance distribution and pretraining compute budget are both matched?

  2. How does SDTA compare with multiple independent corruption draws per input for TimeDART or masked-reconstruction baselines, implemented with linear-memory scaling?

3 comments
AuthorsOpenReview ↗

Response to Reviewer z8ox (Part 1/2)

Thank you for a review that put its finger on the two confounds we needed to resolve. The general response summarizes the new experiments, and here we answer each point with the specific evidence.

W1: forcing one target to s = 1 confounds the count of targets with the distance distribution, and no-forced was only evaluated at W = 4. You are right, and the confound turned out to matter more than we expected. We ran the all-random variant (every target drawn from U{1..12}) at W ∈ {1, 2, 4} for 50 epochs on ten datasets (Table 3, Appendix I.1):

10-ds MSEW=1W=2W=4
forced s=1 (paper)0.34020.33080.3288
every target random0.33220.33110.3302

At W = 1 the paper's protocol leaves a single target that is always the s = 1 window, so the "W = 1" baseline had no distance diversity at all. Removing the forced target, at identical compute, recovers 70 % of the W = 1→4 gap (63–98 % on the three PEMS datasets and electricity). With the distribution matched, the count effect that remains is 0.0020 on the 10-ds mean and 0.005 on PEMS, in W = 4's favor on 8/10 datasets. The paper now recommends the all-random variant as the W = 1 baseline and keeps the forced target as an option (it is a small advantage only on ETTh1). Section 5.2 and Table 3 were rewritten around this result.

W2: epochs rather than compute are matched, and compute-matched runs and repeated-noise/mask controls for the single-window baselines are needed. Three controls, all in Table 3 / Appendix I.1–I.2:

  • Compute. W = 1 (forced) trained for 200 epochs, four times the budget, stays at the W = 1 level: 0.3399 vs 0.3402. Extra epochs alone cannot rescue a single fixed-distance target.
  • Repeated noise inside SDTA. W = 4 with s_max = 1, i.e. four independent noise draws of the same target window, equals W = 1 on all ten datasets (0.3404). Noise averaging is excluded.
  • Baseline budget. TimeDART and TimeMAE-CI pretrained for 150 instead of 50 epochs move by −0.0012 and +0.0016 on the 10-ds mean (TimeMAE-CI gets worse on PEMS). The baselines were not under-budgeted.
  • The same controls under transfer. With a tenth of the target's training windows on PEMS03/04/08, the encoder pretrained on another traffic network and every run trained to convergence (new Figure 3, Table 57), W = 4 ends at 0.3638 and W = 1 at 0.3848. W = 1 with the forced target and four times the pretraining epochs stays at the W = 1 level (0.3889), and W = 1 with a random distance and four times the epochs reaches 0.3531: the pattern of Table 3 again. TimeDART, the same diffusion objective on a single window, ends at 0.3877.

What we could not run in the discussion period are the repeated-corruption variants of TimeDART and TimeMAE-CI themselves (V independent noise or mask draws per input, stacked along the batch). We state this in the revision rather than claim it. The s_max = 1 control answers the "more draws of one target" reading for our own pretext, and the budget control answers "more passes over the data" for the baselines, but a V-draw baseline remains open and we list it as such.

W3: suspected AI polishing ("therefore" 18 times). We removed every occurrence of "therefore" from the main text and appendix (18 → 0) and other connectives of the same kind. The AI use statement already disclosed AI assistance for readability editing, and we have extended it to cover the discussion phase. AI tools suggested some of the control experiments, launched and aggregated the runs, generated the new tables and Figure 3 with scripts, and drafted the revised text and these responses in English from our instructions. The decisions on what to run, what to report and what to concede, including the unfavorable results below, and the final wording are the authors'.

W4: rough layout, single-word lines. Fixed in the revision (the lines you cite and the others we found).

AuthorsOpenReview ↗

Response to Reviewer z8ox (Part 2/2)

Q1: Does W = 4 still beat W = 1 when both the distance distribution and the compute are matched? The answer is no, and the result changed how we describe the axis. With every target random and the number of distance-shifted targets matched (W × epochs = 200), W = 1 for 200 epochs reaches 0.3243, below the paper's W = 4 at 50 epochs (0.3288) and below W = 4 random at 50 epochs (0.3302), by 0.013–0.024 on the three PEMS datasets. W = 2 for 100 epochs reproduces W = 4 for 50 epochs to the fourth decimal (0.3288). So once a random-distance target exists, W and epochs are interchangeable: what the axis scales is the number and diversity of distance-shifted targets, and W is their cheapest carrier rather than a unique source of gain. We still think the axis is worth having, for a cost reason: W = 4 for 50 epochs costs 0.26–0.66× the pretraining time of W = 1 for 200 epochs on the four datasets with timing (Table 36), because the anchor encoding and data loading are shared by the W targets while extra epochs repeat both. The abstract, contributions, Section 5.2 and Section 7 now say this.

Q2: How does SDTA compare with multiple independent corruption draws per input for TimeDART or masked baselines, at linear memory? Inside SDTA this is the s_max = 1 control above (four draws of one target = W = 1, 0.3404 vs 0.3402), which says that the multi-window gain is not the draw count. For TimeDART and TimeMAE-CI we did not run the V-draw variants in time and do not claim the result. The revision lists it as an open control (Section 7).

Positive signals we would like to point to.

  • Pretraining pays once both sides are trained to convergence (Section 6, new Figure 3, Tables 57–59). On PEMS03/04/08 the pretrained encoder ends 5.2 % below the same encoder trained from scratch for as many total epochs with full data (0.2950 vs 0.3112), and 8–10 % below with 5–25 % of the data and an encoder pretrained on another traffic network, on every dataset–seed pair. Pretrained on electricity or weather it keeps about half that gain.
  • The most controlled comparison in the study is the same-seed pair that differs only in the distance distribution (forced W = 1 at 200 epochs, 0.3399, vs random W = 1 at 200 epochs, 0.3243): it isolates distance-shifted targets as the active ingredient of the pretext, which is the mechanism the paper proposes, even though it is not W-specific.
  • All paper rows reproduce on a different machine within 0.0003 (Table 39).
  • The shared pretraining learning rate did not favor SDTA: it is the best of three for TimeMAE-CI and below the best for SDTA (Table 44).
  • At a second stretched horizon (pred_len 1440, PEMS 192) SDTA W = 4 keeps the best 9-dataset mean (0.5640), W = 1 trailing by 0.0294 (Holm p = 0.047).

We also report, in Table 2, that under one fine-tune schedule for every method TimeMAE-CI is ahead by 0.0037, and in Section 6 that under transfer no pretext is best on both PEMS and ETT: TimeMAE-CI leads on PEMS and TimeDART on ETT, each last on the other, and SDTA is second on both. The paper's claims are about what the multi-window axis scales and about this robustness, not about a win over TimeMAE-CI. We would be grateful if you could reconsider the score in light of the controls you asked for, which are now in the paper with the answers they gave.

Reviewer_z8oxOpenReview ↗

Response

I thank the authors for their detailed response and for revising the manuscript so extensively. The revision amounts to an almost complete rewrite, and I appreciate the care the authors took in addressing my earlier comments. I found the paper easier to follow in places, although the overall improvement in clarity was modest given the scale of the revision.

I would encourage the authors to look closely at how good empirical papers in this area present their experiments. Much of the revised section walks the reader through differences in the fourth decimal place of a metric. What I wanted to understand was why predicting windows at different temporal distances helps, when that benefit is largest, and which results support the authors’ interpretation. Without that analysis, parts of the section read much like an agent-generated reproduction report. I also found the discussion of prior work difficult to follow, and the most recent references I could find date to 2024. The presentation issues extend beyond the examples I raised in my initial review and still make it difficult to assess the method’s novelty and the significance of its gains.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response to all reviewers (Part 1/4)

We thank the three reviewers for careful and consistent reviews. The paper is a comparison of self-supervised pretraining pretexts at matched capacity, and the revision says so more plainly. The reviewers' central concerns overlap, so we ran one set of controlled experiments to answer them and report the results here. The individual responses point into this set. All changes in the revised PDF are marked in blue, sentence by sentence against the submitted version. The new Figure 3 and Appendix I are blue throughout, and table numbers in the original appendix are shifted by two because of the two new main-text tables. All new numbers are in the revised PDF (Table 2, Table 3, Figure 3 and Appendix I) and were produced on RTX 4090 machines (PyTorch 2.14) with the paper's fixed configuration and five seeds, on ten of the twelve datasets (PEMS07 and traffic were excluded for cost). The paper's own rows reproduce there within 0.0003 on the 10-dataset mean (Appendix I, Table 39), so old and new rows are directly comparable.

What we ran

Question raisedExperimentWhere
Is the gain from W just more compute? (z8ox W2/Q1, pu7N W2/Q2, jf9A W3/Q2)W=1 pretrained for 200 epochs (4× the budget), W=2 for 100 epochs (iso-target), baselines pretrained for 150 epochsTable 3, App. I.1 and I.2
Is it just repeated noise? (z8ox W2/Q2, pu7N W1/Q1)W=4 with s_max=1: four noise draws of the same targetTable 3
Does forced s=1 confound count with distance distribution? (z8ox W1/Q1)Every target at a random distance, W∈{1,2,4}, 50 epochs, and W=1 random for 200 epochsTable 3, App. I.1
Are the baselines fine-tuned fairly / well tuned? (pu7N W2)One fine-tune schedule for every method (OneCycleLR for all, and step-halving for all), and a pretraining learning-rate sweep {3e-4, 1e-3, 3e-3}Table 2, App. I.2 and I.3
Why pretrain at all? (jf9A W2/Q1, pu7N Q1)The same encoder trained directly on the forecasting task: for the 60-epoch total budget (ten datasets), and with both sides trained to convergence on PEMS03/04/08 (pretraining + 60 fine-tune epochs vs 110 and 260 epochs from scratch), with full and with scarce dataSection 6, Figure 3, App. I.4 and I.8
Does the gain hold with larger encoders? (jf9A W4/Q4)Width d∈{32,64,128} at W∈{1,4}, with TimeMAE-CI and TimeDART at d=128App. I.5
Generalization: longer inputs, longer horizons, distribution shift, scarce data (pu7N W4/Q4, jf9A Q1)Input length 336, second extended horizon (pred_len 1440, PEMS 192), cross-dataset transfer (ETTh1↔ETTh2, ETTm1↔ETTm2, PEMS04→PEMS08), and transfer to PEMS03/04/08 with 5 / 10 / 25 % of the target windows from another traffic network, electricity and weather, and to the four ETT datasets with 10 %, for SDTA, TimeMAE-CI and TimeDARTApp. I.6–I.7, Figure 3, App. I.8 (Tables 56–60)

What we found: the mechanism claim holds, stated more carefully

(i) Not compute. W=1 with the paper's forced s=1 target trained for 200 epochs stays at the W=1 level (0.3399 vs 0.3402 at 50 epochs, against 0.3288 for W=4 at 50 epochs). TimeDART and TimeMAE-CI pretrained for 150 instead of 50 epochs move by −0.0012 and +0.0016. (ii) Not repeated noise. W=4 with s_max=1 equals W=1 on all ten datasets (0.3404). (iii) The paper's W=1 baseline had zero distance diversity. Its single target is always the forced s=1 window. Letting that one target take a random distance, at identical compute, recovers 70 % of the W=1→4 gap (0.3402 → 0.3322). With every target random, the count effect that remains is 0.0020 on the 10-ds mean (0.005 on PEMS), still in W=4's favor on 8/10 datasets. (iv) Once a random-distance target exists, W and epochs are interchangeable. W=2 for 100 epochs reproduces W=4 for 50 epochs to the fourth decimal (0.3288), and W=1 random for 200 epochs reaches 0.3243, below every 50-epoch configuration. The ingredient the axis scales is the number and diversity of distance-shifted targets. W is the cheapest way to add them at a fixed epoch budget: W=4 for 50 epochs costs 0.26–0.66× the pretraining time of W=1 for 200 epochs on the four datasets of Table 36.

AuthorsOpenReview ↗

General response to all reviewers (Part 2/4)

Three findings go against the submitted narrative, and the paper now states them

(a) Under one fine-tune schedule for every method, TimeMAE-CI is ahead. The submission fine-tuned each method as its own paper does (OneCycleLR for SDTA and TimeDART, step-halving for TimeMAE-CI, CPC, SimTS). OneCycleLR helps every baseline by 0.005–0.008. TimeMAE-CI moves from 0.3297 to 0.3251 and edges ahead of SDTA (0.3288) on 6/10 datasets (Wilcoxon p = 0.49), and under step-halving for all, SDTA (0.3329) also trails TimeMAE-CI (0.3297). Table 1 is unchanged. New Table 2 reports the unified protocol, and the text now claims parity under the paper's protocol and second place under the unified one. (b) The submission's evidence that pretraining helps was a 10-epoch control, and it does not survive. The same encoder trained from scratch for 60 epochs, the total of pretraining and fine-tuning, reaches 0.3296 vs 0.3288 for pretrain-then-fine-tune on ten datasets. We no longer use the 10-epoch control. Neither side of that comparison has converged, though. Part 3 gives the comparison with both sides trained to convergence, and it favors pretraining. (c) The gain from W is conditional. It concentrates on short inputs and high-channel, periodic data. At input length 336 it vanishes on the six low-channel datasets (W=1 0.3397 vs W=4 0.3401). It shrinks with width on those datasets (0.0040 → 0.0025 → 0.0016 at d = 32/64/128) while growing on PEMS08 (0.020 → 0.030 → 0.050). Doubling the width at W=1 buys as much as W=1→4 at d=32, and at d=128 the W=4 model is the best eight-dataset row (0.3223, against 0.3318 for TimeMAE-CI and 0.3326 for TimeDART at the same width).

Positive results we would like to highlight

  • Pretraining pays once both sides are trained to convergence, and it transfers (Section 6, new Figure 3, Tables 57–59, details in Part 3). On PEMS03/04/08 the pretrained encoder ends 5.2 % below the same encoder trained from scratch for as many total epochs with full data (0.2950 vs 0.3112). With 5–25 % of the data and an encoder pretrained on another traffic network it ends 8–10 % below, on every dataset–seed pair. Pretrained on electricity or weather it keeps about half the gain.
  • SDTA is the pretext that transfers robustly. Under transfer SDTA is second on both PEMS and ETT, at most 3.0 % behind the best, while TimeMAE-CI and TimeDART are each last on one family by 8.6–10.1 % (Tables 57 and 60).
  • A controlled pair isolates the mechanism. Same W, same epochs, same seeds, only the distance distribution differs: forced-s=1 W=1 at 200 epochs 0.3399 vs random-distance W=1 at 200 epochs 0.3243, with gaps of 0.04–0.06 on the three PEMS datasets. Distance-shifted targets are the active ingredient of the pretext.
  • Reproducibility. All five re-pretrained paper rows (SDTA W=1/2/4, TimeMAE-CI, TimeDART) reproduce on a different machine and PyTorch version within 0.0003 on the 10-ds mean (Table 39).
  • The shared learning rate did not favor SDTA. For TimeMAE-CI the shared 1e-3 is the best of {3e-4, 1e-3, 3e-3}, and the alternatives cost it 0.0088 and 0.0072. TimeDART is insensitive (±0.001). SDTA itself would gain 0.0038 at 3e-3, better on six of nine datasets (Table 44).
  • Longer horizons. At pred_len 1440 (PEMS 192) on nine datasets SDTA W=4 keeps the best mean (0.5640). W=1 trails by 0.0294 (Holm p = 0.047), TimeDART and CPC by ≈0.028, SimTS by 0.103, TimeMAE-CI by 0.0041 (Table 54). The W gain at this horizon is 0.074 on PEMS.
  • Width. At d=128, W=4 is the best eight-dataset row, about 0.01 ahead of TimeMAE-CI and TimeDART at the same width (Table 50).
  • Cost. Everything above was produced at the paper's capacity (17k transferred parameters) on consumer GPUs. W adds no deployment cost, and fine-tuning time is flat in W.
AuthorsOpenReview ↗

General response to all reviewers (Part 3/4)

Pretraining against training from scratch, both to convergence (Figure 3, Appendix I.8, Tables 56–60)

Why a new protocol. The paper's 10 fine-tune epochs are a budget for the full training split. With a tenth of the data they are a tenth of the gradient steps, and at least four of five such fine-tunes end with the last epoch as their best. A comparison at that budget compares schedules, not end points. In the new protocol the pretrained encoder is fine-tuned for 60 epochs, and the same encoder is trained from scratch for 110 epochs (the same total as 50 pretraining plus 60 fine-tune epochs) and for 260. All runs use the downstream schedule and early stopping (patience 10). Nearly all 260-epoch runs stop early, so the from-scratch end point is reached.

PEMS03/04/08, 3-ds MSE, 5 seedsfrom scratch, 110 ep.from scratch, 260 ep.SDTA W=4TimeMAE-CITimeDART
5 % of the target windows, encoder from another traffic network0.48700.45660.44890.44810.4638
10 %0.40580.39540.36380.35710.3877
25 %0.3525–0.31950.30750.3430
full split, encoder pretrained on the target0.3112–0.2950––

(a) The pretrained end point is lower: −5.2 % with full data and −7.8 / −10.3 / −9.3 % at 5 / 10 / 25 %, on every dataset–seed pair (60 of 60). At 10 % the 260-epoch from-scratch run is still 8.0 % behind (1.7 % at 5 %). (b) The gain grows with the horizon: none at prediction length 12, 11–15 % at 96 (Table 59). (c) The source can be another domain: pretrained on electricity or weather the encoder ends at 0.3833 and 0.3876 at 10 %, about half the cross-network gain, ahead of training from scratch on every pair. (d) Pretexts: W=4 ends 5.5 % below W=1 (0.3848) and 6.2 % below TimeDART. Given four times the pretraining epochs, W=1 stays at its level with the forced target (0.3889) and reaches 0.3531 with a random distance, the pattern of Table 3 under transfer. Against its own encoder trained from scratch SDTA gains the most (10.3 %, against 7.4 % for TimeMAE-CI and 5.1 % for TimeDART). (e) No pretext transfers best everywhere, and SDTA never falls far behind. We ran the same protocol for all three pretexts on the four ETT datasets at 10 % (Table 60). There the three encoders start level from scratch (0.4500 / 0.4523 / 0.4501), and no pretext pays (SDTA +3.5 %, TimeMAE-CI +10.0 %, TimeDART +0.4 %). The order flips between the two families. On PEMS TimeMAE-CI is best (0.3571), with SDTA 1.9 % and TimeDART 8.6 % behind. On ETT TimeDART is best (0.4520), with SDTA 3.0 % and TimeMAE-CI 10.1 % behind. SDTA is second on both, the smallest worst case of the three. (f) Limits: the convergence comparison with full data covers the three PEMS benchmarks.

Cross-dataset transfer at the paper's fine-tune (ETTh1↔ETTh2, ETTm1↔ETTm2, PEMS04→PEMS08, full target split, Table 56): the encoder ports, with transfer W=4 0.0035 behind in-dataset W=4 on the five-dataset mean and transfer W=1 equal to in-dataset W=1. The W gain survives the shift at about 60 % of its size.

AuthorsOpenReview ↗

General response to all reviewers (Part 4/4)

Summary of changes in the revised PDF

Changed or added text is set in blue in the uploaded PDF, so the diff against the submission can be read directly. Unchanged sentences stay black.

  • Abstract, introduction (contributions), Section 8. Claims are restated conditionally, and "same training budget" becomes "same pretraining epoch budget". Numbers and defensive statements are removed from the introduction. Every term is defined at first use (anchor, target, shift s, s_max, supervision axis, past-link mask ratio m, capacity-matched). The convergence and transfer results appear in the abstract, the introduction and the conclusion.
  • Section 2. One clause per baseline says what it reconstructs and how SDTA differs. TS2Vec, CoST and AutoCon are added, and the "cross-window" claim is restricted to generative, masked and diffusion pretexts.
  • Section 3. The overlap for s < 12 and s_max = 12 is stated. The index-based cross-attention mask is written out: every target patch is predicted from context ending ≥ s patches before it. Instance-normalization is made explicit, and Algorithm 1 is indexed per patch, with m and s_min in its parameter list.
  • Section 5. 5.1: new Table 2 (unified fine-tune schedule) and a corrected reading of Table 1. 5.2: new Table 3 and the "What the axis buys" paragraph. 5.3: baseline budget and learning-rate checks. 5.4: the cost argument for W. 5.5: second extended horizon. 5.6: the MAE exceptions (TimeMAE-CI and CPC) and the extended-horizon counts corrected.
  • Sections 6 and 7. The no-diffusion arm is identified as a clean-query shortcut. New paragraphs and Figure 3 compare pretraining with training from scratch, both trained to convergence, with full data and with scarce data under transfer, for three pretexts on PEMS and ETT. Section 7 states the conditional nature of the W gain, second place under the unified protocol, and the limits of the convergence and transfer evidence.
  • Appendix I (new). Protocol, reproduction, and all tables behind Sections 5–7. Appendix tables 6–38 keep their order but are renumbered by +2 because of the two new main-text tables (the submission's Table 34 is now Table 36, etc.).
  • Writing. The introduction and abstract are rewritten: every term defined at first use, no result numbers in the introduction, no sentence over 35 words, and TimeMAE-CI named instead of "the runner-up". The word "therefore" (18 occurrences in the submission), similar connectives, and most clause-joining semicolons are removed throughout. One-word paragraph-final lines and the title's hyphen break are fixed. The table and caption corrections are listed in the individual responses, and the AI use statement now covers the discussion phase: the control experiments that AI tools suggested, the runs they launched, and the tables, figure and text they drafted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.