- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a systematic empirical study of when self-supervised learning (SSL) provides transferable representations for time-series tasks. The authors benchmark seven representative SSL methods spanning five paradigms—reconstruction, generative modeling, invariance-based learning, contrastive learning, and joint-embedding predictive architectures, using a shared Transformer encoder to control architectural differences. The evaluation covers three downstream tasks, including forecasting (7 datasets), classification (9 datasets), and anomaly detection (5 datasets). To assess the benefits of pre-training, the authors compare linear probing, MLP probing, and full fine-tuning against matched non-pre-trained baselines. They further investigate the effects of pre-training data sources (real-world, synthetic, and in-domain), data scale, training duration, and encoder depth, supplemented by ablation studies and representation analyses.
The experimental findings indicate that SSL transferability is highly task-dependent: pre-training generally benefits anomaly detection, provides selective improvements for classification, particularly under fine-tuning, but offers limited advantages over non-pre-trained baselines for forecasting. The authors also observe that no single SSL method consistently dominates across tasks, linear probing performance does not reliably predict fine-tuning outcomes, and anomaly detection rankings depend on evaluation metrics. Furthermore, synthetic pre-training can be competitive with real-world data, while increasing pre-training data volume, training duration, or encoder depth does not consistently improve transfer performance under the evaluated settings. Based on these findings, the paper provides practical recommendations for evaluating and developing time-series SSL methods.
Critical strengths
-
A systematic and well-controlled evaluation of time-series SSL transferability. The paper addresses an important and insufficiently examined question: whether self-supervised pre-training consistently improves downstream time-series performance under a common backbone architecture and comparable training settings. By evaluating seven SSL methods spanning five paradigms across forecasting, classification, and anomaly detection, the authors provide a relatively comprehensive empirical comparison. I particularly appreciate the inclusion of matched randomly initialized baselines for both frozen probing and full fine-tuning, as well as the additional equal-hyperparameter experiments in Appendix F.2. These controls make the evaluation more informative than comparisons that attribute improvements to pre-training without accounting for architectural differences. This is a meaningful contribution to establishing more rigorous evaluation practices for time-series representation learning.
-
Informative findings that challenge common assumptions about SSL transfer. The paper provides several practically consequential observations. In particular, the substantially different transfer behaviors across downstream tasks, the discrepancy between linear probing and fine-tuning rankings, and the importance of pre-training data relevance offer valuable insights into the limitations of universal time-series representations. The finding that pre-training frequently fails to outperform supervised training from scratch on forecasting tasks is particularly noteworthy. The comparison of multiple anomaly detection metrics also demonstrates that conclusions can depend strongly on evaluation choices. These findings, together with the detailed supplementary experiments, provide useful guidance for future SSL research even without introducing a new learning algorithm.
Critical weaknesses
-
Limited clarity regarding potential overlap between pre-training and downstream datasets. The paper does not clearly establish whether potential overlaps between the Monash pre-training corpus and downstream datasets have been systematically examined. Since the study focuses on evaluating the transferability of SSL across different tasks, a clearer description of the pre-training data composition and overlap checks would strengthen the reliability and interpretation of the experimental findings.
-
Limited experimental coverage and consistency in some evaluations. The depth-scaling experiments are primarily conducted on three forecasting datasets with a single random seed, which limits the strength of the conclusions regarding model depth. In addition, the forecasting experiments use different dataset sets across adaptation protocols, with Electricity and Traffic excluded from MLP probing and fine-tuning. These limitations make it more difficult to assess the generality of the depth-scaling observations and to directly compare transfer performance across adaptation protocols.
Questions for authors
I encourage the authors to address some of the following issues in the rebuttal and I would adjust the rating based on the results.
-
Since the composition of pre-training data is important for evaluating SSL transferability, could the authors clarify whether they have checked for overlap between the Monash pre-training corpus and the downstream datasets, including their training and test splits? In particular, how is potential data overlap handled when interpreting cross-domain transfer results?
-
The depth-scaling experiments mainly focus on three forecasting datasets and use a single random seed. While Appendix E.1 includes additional classification results with a 24-layer model, could the authors clarify the rationale for the limited scope of the full depth sweep and whether similar trends hold for other downstream tasks? Additionally, were the learning rates and other optimization hyperparameters adjusted for different model depths?
-
In Table 2, the forecasting evaluation includes 49 objective–dataset pairs under linear probing but only 35 under MLP probing and fine-tuning. Could the authors clarify why Electricity and Traffic were excluded from the latter two settings? A comparison using a consistent set of datasets across adaptation protocols would also be helpful.
-
The following minor issues do not affect my overall assessment or rating of the paper, but I would appreciate it if the authors could confirm them:
In Appendix A.3 (line 680), "View Defections" might be a typo. Did the authors mean "View Definitions"?
In Appendix D.2.2 (line 1481), the table reference appears as "Table ??".
3 comments
Question 4
We corrected “View Defections” to “View Definitions” in Appendix A.3 and replaced “Table ??” with “Table 21” in Appendix D.2.2, referring to the in-domain forecasting results.
We thank the reviewer again and hope these additions address their concerns. We would be glad to clarify anything further, and if the points raised are resolved, we hope the reviewer will consider raising their score.
The authors have addressed my main concerns, and I have raised my rating accordingly. Good luck.
Response to reviewer
We thank the reviewer for the thoughtful assessment and recognition of our controlled evaluation and practical findings. The constructive suggestions helped us clarify pre-training data overlap, expand the experimental coverage, and improve consistency across evaluation protocols. These additions, clarifications, and corrections are included in the revised paper.
Question 1
Our Monash pre-training corpus contains 22 files. We checked every file against the downstream benchmarks. Traffic data was removed from the corpus previously, so the only overlap is with Electricity. Monash's weather file is Australian daily weather and is unrelated to the downstream Weather dataset. No ETT, anomaly-detection, or classification data appears in the corpus.
This mirrors the overlap in prior work that pre-trains on Monash and evaluates on Electricity, such as MOMENT ([Goswami et al., ICML 2024]) and Moirai ([Woo et al., ICML 2024]). TTM ([Ekambaram et al., NeurIPS 2024] instead removes Electricity, Weather and Traffic from its Monash pre-training data; we follow it in removing Traffic, and quantify the effect of the remaining Electricity overlap below.
To measure the effect of the overlap, we re-pre-trained all seven objectives on Monash with Electricity removed and re-evaluated on Electricity (linear probing - single seed). We find that Removing Electricity from pre-training has negligible effect on the result across all objectives (see Tables 22 in the revised Appendix). NTP and DINO change by less than 0.002 MSE, and the other five are equal or slightly better (mean improvement 0.008 ) without it. The overlap therefore does not inflate our Electricity results. It also does not produce positive transfer: even with the Electricity series in the pre-training data, no objective beats the random frozen encoder (0.172). The same holds across pre-training sources (Tables 10 and 21–24 in the revised Appendix).
Question 2
We expanded the forecasting depth sweep to all seven datasets with two seeds (Figure 16 in the revised Appendix, pdf attached). Due to compute constraints, some Electricity and Traffic configurations currently have a single seed; the second seed will be added in the camera-ready version. Depth does not improve forecasting: the best depth is 2–4 layers in 30 of 49 objective–dataset pairs and 24 layers in only 10. Going from 8 to 24 layers raises MSE on most datasets for five of seven objectives (MAE: worse on all seven, +4.5%). Diffusion is the only objective that benefits (−3.4%, better on five of seven).
Following the suggestion of Reviewer TdZ7, and to understand why, we probed the frozen 24-layer encoder after its first k ∈ {2, 4, 8, 12, 24} blocks, with a separate linear head per k (Figure 18, Appendix E.2). An intermediate exit beats the final layer in 42 of 49 pairs, usually after 2–4 blocks, so forecasting-relevant information peaks early. This matches findings for pre-trained vision and language models (Chen et al., 2020; El-Nouby et al., 2024; Skean et al., 2025; Uselis & Oh, 2025; Halawi et al., 2024).
For other tasks, a new anomaly-detection depth sweep (Figure 17) shows no degradation with depth: JEPA, LeJEPA, Diffusion and SoftCLT are nearly flat, and NTP, MAE and DINO improve on MSL and SMAP. In classification, the 8-layer encoder beats the 24-layer one in 31 of 63 pairs vs. 19 (Appendix E.1).
Learning rates were scaled with depth for stability (×6, ×2, ×1, ×0.4, ×0.2 for 2, 4, 8, 12, 24 layers; values in Appendix E.2); all other hyperparameters are unchanged.
References
Question 3
MLP probing and finetuning were originally reported for 35 pairs due to time and compute constraints. The revised manuscript (pdf attached) now includes the additional MLP probing and full fine-tuning experiments on the Electricity and Traffic, across all objectives and their matched random controls. These additional experiments show similar trends to the ones previously reported. With an MLP probe, no objective beats the random frozen encoder on Traffic; on Electricity, only Diffusion does, by 0.001 MSE. Under fine-tuning, supervised training from scratch outperforms every objective on both datasets. Over the full set of 49 pairs, SSL beats the baseline in 12/49 (MSE) and 11/49 (MAE) cases under MLP probing, and in 3/49 and 2/49 cases under fine-tuning. Across all three protocols, forecasting therefore remains the task where pre-training helps least. We updated Table 2 and Appendices D.1.2 (Table 15) and D.1.3 (Table 18) accordingly.