acceptodds
Under review as a conference paper at ICLR 2027

When Does Self-Supervised Learning Transfer to Time-Series Tasks?

41 views

Abstract

Self-supervised learning (SSL) assumes that solving pretext tasks on unlabeled data yields representations that transfer effectively across downstream applications via linear probing or fine-tuning. While this paradigm has driven major progress in vision and language, its benefits for time series remain under-investigated and often confounded by inconsistent experimental controls. To address this gap, we benchmark seven representative methods from five key SSL paradigms across anomaly detection, classification, and forecasting under parameter- and data-matched budgets. We find that transfer efficacy depends heavily on the downstream task. SSL yields substantial gains in anomaly detection and provides effective initializations for classification under fine-tuning, but offers limited to no advantage over non-pre-trained controls in forecasting. Furthermore, linear probing does not reliably predict fine-tuning performance, synthetic pre-training is often competitive with real-world corpora, and scaling encoder depth degrades forecasting accuracy. We synthesize these empirical results into practical evaluation and development guidelines for time-series SSL. Our code is available in the supplementary material.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series, Self supervised learning, Representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UckSOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a systematic empirical study of when self-supervised learning (SSL) provides transferable representations for time-series tasks. The authors benchmark seven representative SSL methods spanning five paradigms—reconstruction, generative modeling, invariance-based learning, contrastive learning, and joint-embedding predictive architectures, using a shared Transformer encoder to control architectural differences. The evaluation covers three downstream tasks, including forecasting (7 datasets), classification (9 datasets), and anomaly detection (5 datasets). To assess the benefits of pre-training, the authors compare linear probing, MLP probing, and full fine-tuning against matched non-pre-trained baselines. They further investigate the effects of pre-training data sources (real-world, synthetic, and in-domain), data scale, training duration, and encoder depth, supplemented by ablation studies and representation analyses.

The experimental findings indicate that SSL transferability is highly task-dependent: pre-training generally benefits anomaly detection, provides selective improvements for classification, particularly under fine-tuning, but offers limited advantages over non-pre-trained baselines for forecasting. The authors also observe that no single SSL method consistently dominates across tasks, linear probing performance does not reliably predict fine-tuning outcomes, and anomaly detection rankings depend on evaluation metrics. Furthermore, synthetic pre-training can be competitive with real-world data, while increasing pre-training data volume, training duration, or encoder depth does not consistently improve transfer performance under the evaluated settings. Based on these findings, the paper provides practical recommendations for evaluating and developing time-series SSL methods.

Critical strengths
  1. A systematic and well-controlled evaluation of time-series SSL transferability. The paper addresses an important and insufficiently examined question: whether self-supervised pre-training consistently improves downstream time-series performance under a common backbone architecture and comparable training settings. By evaluating seven SSL methods spanning five paradigms across forecasting, classification, and anomaly detection, the authors provide a relatively comprehensive empirical comparison. I particularly appreciate the inclusion of matched randomly initialized baselines for both frozen probing and full fine-tuning, as well as the additional equal-hyperparameter experiments in Appendix F.2. These controls make the evaluation more informative than comparisons that attribute improvements to pre-training without accounting for architectural differences. This is a meaningful contribution to establishing more rigorous evaluation practices for time-series representation learning.

  2. Informative findings that challenge common assumptions about SSL transfer. The paper provides several practically consequential observations. In particular, the substantially different transfer behaviors across downstream tasks, the discrepancy between linear probing and fine-tuning rankings, and the importance of pre-training data relevance offer valuable insights into the limitations of universal time-series representations. The finding that pre-training frequently fails to outperform supervised training from scratch on forecasting tasks is particularly noteworthy. The comparison of multiple anomaly detection metrics also demonstrates that conclusions can depend strongly on evaluation choices. These findings, together with the detailed supplementary experiments, provide useful guidance for future SSL research even without introducing a new learning algorithm.

Critical weaknesses
  1. Limited clarity regarding potential overlap between pre-training and downstream datasets. The paper does not clearly establish whether potential overlaps between the Monash pre-training corpus and downstream datasets have been systematically examined. Since the study focuses on evaluating the transferability of SSL across different tasks, a clearer description of the pre-training data composition and overlap checks would strengthen the reliability and interpretation of the experimental findings.

  2. Limited experimental coverage and consistency in some evaluations. The depth-scaling experiments are primarily conducted on three forecasting datasets with a single random seed, which limits the strength of the conclusions regarding model depth. In addition, the forecasting experiments use different dataset sets across adaptation protocols, with Electricity and Traffic excluded from MLP probing and fine-tuning. These limitations make it more difficult to assess the generality of the depth-scaling observations and to directly compare transfer performance across adaptation protocols.

Questions for authors

I encourage the authors to address some of the following issues in the rebuttal and I would adjust the rating based on the results.

  1. Since the composition of pre-training data is important for evaluating SSL transferability, could the authors clarify whether they have checked for overlap between the Monash pre-training corpus and the downstream datasets, including their training and test splits? In particular, how is potential data overlap handled when interpreting cross-domain transfer results?

  2. The depth-scaling experiments mainly focus on three forecasting datasets and use a single random seed. While Appendix E.1 includes additional classification results with a 24-layer model, could the authors clarify the rationale for the limited scope of the full depth sweep and whether similar trends hold for other downstream tasks? Additionally, were the learning rates and other optimization hyperparameters adjusted for different model depths?

  3. In Table 2, the forecasting evaluation includes 49 objective–dataset pairs under linear probing but only 35 under MLP probing and fine-tuning. Could the authors clarify why Electricity and Traffic were excluded from the latter two settings? A comparison using a consistent set of datasets across adaptation protocols would also be helpful.

  4. The following minor issues do not affect my overall assessment or rating of the paper, but I would appreciate it if the authors could confirm them:

    In Appendix A.3 (line 680), "View Defections" might be a typo. Did the authors mean "View Definitions"?

    In Appendix D.2.2 (line 1481), the table reference appears as "Table ??".

3 comments
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for the thoughtful assessment and recognition of our controlled evaluation and practical findings. The constructive suggestions helped us clarify pre-training data overlap, expand the experimental coverage, and improve consistency across evaluation protocols. These additions, clarifications, and corrections are included in the revised paper.

Question 1

Our Monash pre-training corpus contains 22 files. We checked every file against the downstream benchmarks. Traffic data was removed from the corpus previously, so the only overlap is with Electricity. Monash's weather file is Australian daily weather and is unrelated to the downstream Weather dataset. No ETT, anomaly-detection, or classification data appears in the corpus.

This mirrors the overlap in prior work that pre-trains on Monash and evaluates on Electricity, such as MOMENT ([Goswami et al., ICML 2024]) and Moirai ([Woo et al., ICML 2024]). TTM ([Ekambaram et al., NeurIPS 2024] instead removes Electricity, Weather and Traffic from its Monash pre-training data; we follow it in removing Traffic, and quantify the effect of the remaining Electricity overlap below.

To measure the effect of the overlap, we re-pre-trained all seven objectives on Monash with Electricity removed and re-evaluated on Electricity (linear probing - single seed). We find that Removing Electricity from pre-training has negligible effect on the result across all objectives (see Tables 22 in the revised Appendix). NTP and DINO change by less than 0.002 MSE, and the other five are equal or slightly better (mean improvement 0.008 ) without it. The overlap therefore does not inflate our Electricity results. It also does not produce positive transfer: even with the Electricity series in the pre-training data, no objective beats the random frozen encoder (0.172). The same holds across pre-training sources (Tables 10 and 21–24 in the revised Appendix).

Question 2

We expanded the forecasting depth sweep to all seven datasets with two seeds (Figure 16 in the revised Appendix, pdf attached). Due to compute constraints, some Electricity and Traffic configurations currently have a single seed; the second seed will be added in the camera-ready version. Depth does not improve forecasting: the best depth is 2–4 layers in 30 of 49 objective–dataset pairs and 24 layers in only 10. Going from 8 to 24 layers raises MSE on most datasets for five of seven objectives (MAE: worse on all seven, +4.5%). Diffusion is the only objective that benefits (−3.4%, better on five of seven).

Following the suggestion of Reviewer TdZ7, and to understand why, we probed the frozen 24-layer encoder after its first k ∈ {2, 4, 8, 12, 24} blocks, with a separate linear head per k (Figure 18, Appendix E.2). An intermediate exit beats the final layer in 42 of 49 pairs, usually after 2–4 blocks, so forecasting-relevant information peaks early. This matches findings for pre-trained vision and language models (Chen et al., 2020; El-Nouby et al., 2024; Skean et al., 2025; Uselis & Oh, 2025; Halawi et al., 2024).

For other tasks, a new anomaly-detection depth sweep (Figure 17) shows no degradation with depth: JEPA, LeJEPA, Diffusion and SoftCLT are nearly flat, and NTP, MAE and DINO improve on MSL and SMAP. In classification, the 8-layer encoder beats the 24-layer one in 31 of 63 pairs vs. 19 (Appendix E.1).

Learning rates were scaled with depth for stability (×6, ×2, ×1, ×0.4, ×0.2 for 2, 4, 8, 12, 24 layers; values in Appendix E.2); all other hyperparameters are unchanged.

References

  • Chen et al. Generative Pretraining from Pixels. ICML 2020.
  • El-Nouby et al. Scalable Pre-training of Large Autoregressive Image Models. ICML 2024.
  • Skean et al. Layer by Layer: Uncovering Hidden Representations in Language Models. ICML 2025.
  • Uselis & Oh. Intermediate Layer Classifiers for OOD Generalization. ICLR 2025.
  • Halawi et al. Overthinking the Truth: Understanding How Language Models Process False Demonstrations. ICLR 2024.

Question 3

MLP probing and finetuning were originally reported for 35 pairs due to time and compute constraints. The revised manuscript (pdf attached) now includes the additional MLP probing and full fine-tuning experiments on the Electricity and Traffic, across all objectives and their matched random controls. These additional experiments show similar trends to the ones previously reported. With an MLP probe, no objective beats the random frozen encoder on Traffic; on Electricity, only Diffusion does, by 0.001 MSE. Under fine-tuning, supervised training from scratch outperforms every objective on both datasets. Over the full set of 49 pairs, SSL beats the baseline in 12/49 (MSE) and 11/49 (MAE) cases under MLP probing, and in 3/49 and 2/49 cases under fine-tuning. Across all three protocols, forecasting therefore remains the task where pre-training helps least. We updated Table 2 and Appendices D.1.2 (Table 15) and D.1.3 (Table 18) accordingly.

Question 4

We corrected “View Defections” to “View Definitions” in Appendix A.3 and replaced “Table ??” with “Table 21” in Appendix D.2.2, referring to the in-domain forecasting results.

We thank the reviewer again and hope these additions address their concerns. We would be glad to clarify anything further, and if the points raised are resolved, we hope the reviewer will consider raising their score.

The authors have addressed my main concerns, and I have raised my rating accordingly. Good luck.

Reviewer_CYd4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper benchmarks seven self-supervised learning methods across time-series forecasting, classification, and anomaly detection using a shared Transformer backbone and matched non-pretrained controls. It evaluates different adaptation protocols, pretraining sources, data scales, and model depths. The results show task-dependent transfer benefits, with stronger gains in anomaly detection and classification after fine-tuning, but limited forecasting improvements.

Critical strengths
  1. Meaningful baseline controls. Comparing pretrained encoders against frozen random features and training from scratch helps distinguish pretraining benefits from architectural and downstream-training effects.

  2. Useful cross-task evaluation. Multiple adaptation protocols and complementary anomaly-detection metrics reveal consequential differences that a single-task or single-metric evaluation would miss.

Critical weaknesses
  1. Adaptation robustness. Fine-tuning sensitivity is insufficiently examined, leaving unclear whether negative forecasting transfer reflects representation quality or adaptation settings.

  2. Limited generality. Conclusions rely on one Transformer architecture; depth-scaling experiments cover only three forecasting datasets and one seed.

  3. Implementation inconsistencies. Conflicting descriptions of NTP window lengths and SIGReg inputs obscure data matching and reproducibility.

  4. Missing architecture diagrams. The relationships between the shared backbone, method-specific modules, and downstream adaptation are difficult to follow without a model diagram.

Questions for authors

See the weaknesses.

Reviewer_TdZ7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the transferability of SSL for time series analysis. It benchmarks seven methods across five SSL paradigms on forecasting, classification, and anomaly detection, using controlled Transformer backbones and multiple downstream evaluation protocols. Experiments show that SSL provides substantial benefits for anomaly detection and selective gains for classification, but limited improvements in forecasting. The paper further analyzes the effects of pre-training data, model depth, and adaptation strategies, offering practical guidelines for evaluating time-series SSL.

Critical strengths
  1. The paper is well-written and clearly organized, making its motivation, methodology, and main findings easy to understand.

  2. The paper systematically evaluates multiple SSL paradigms across downstream tasks, using matched baselines and different adaptation protocols to provide meaningful comparisons.

  3. The findings highlight important limitations of time-series SSL transferability and provide useful empirical guidance for future research.

Critical weaknesses
  1. The distinctions among different pre-training settings, evaluation protocols, and experimental configurations are not always sufficiently clear, making some comparisons difficult to follow.

  2. Some paradigms are represented by only one method, which may introduce bias and limit the validity of conclusions drawn at the paradigm level.

  3. Evaluating only Transformer backbones restricts the generalizability of the findings. Moreover, some experimental observations receive relatively superficial explanations, while certain conclusions appear overly broad given the available evidence.

Questions for authors
  1. Since there is substantial variation in SSL transferability across tasks and pre-training sources, under what conditions do different SSL objectives transfer effectively, and how do task-specific and domain-dependent factors affect the generalizability of the findings?
  2. As the authors claim that deeper encoders often degrade forecasting performance in the paper, how does task-relevant information evolve across layers? Could layer-wise probing reveal whether intermediate representations are more suitable for certain downstream tasks?
  3. Could the performance degradation in deeper models be attributed to later layers suppressing useful information from earlier layers? Would analyzing layer-wise predictions help identify potential overthinking and explore the feasibility of early-exit strategies?
  4. Since some SSL paradigms are represented by only a single method, such as SoftCLT for contrastive learning, to what extent can the findings be generalized to the entire paradigm rather than being specific to the selected methods?
  5. All SSL methods in this paper are evaluated using a Transformer backbone. To what extent can the findings generalize to other architectures, such as CNNs? Could the observed transfer patterns be architecture-dependent?

Comments on the paper

1 comment
Area_Chair_DybYOpenReview ↗

Discussion period ending soon

Dear Reviewers,

Thank you for your reviews of this submission. The authors have posted responses addressing the concerns raised.

With limited time remaining in the discussion period, please take a moment to review the authors' responses and post a brief follow-up indicating whether your concerns have been addressed or what remains outstanding. If your assessment changes as a result, please update your score accordingly.

Active engagement at this stage is essential for a fair and well-substantiated final decision. Thank you for your time and effort in reviewing this submission.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.