- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors propose a self-supervised pretraining method for time-series classification models using a simple pretext task. They evaluate their pretraining framework under a simple evaluation setup with the InceptionTime architecture and compare it against several SSL baselines.
Critical strengths
The proposed method is very compute-efficient during both pretraining and inference, which also enables easy deployment.
Critical weaknesses
1. Outdated Evaluation Setup (Model Architecture, Baselines, Benchmarks)
The proposed pretraining framework is evaluated in an outdated evaluation scheme that does not follow recent literature. Specifically:
- Model architecture: The proposed model and all baselines are evaluated using the InceptionTime architecture. This architecture does not model temporal interactions but instead uses global average pooling over patch embeddings. This is an extremely restricted setup and ignores modern architectures that include temporal interactions (CNNs and transformers). It is therefore unclear whether the proposed pretraining framework provides any benefits when using an architecture with temporal interactions at inference.
- Baselines: The authors do not compare against any modern time-series foundation models and also lack common simple time-series baselines. Missing baselines include:
- UniTS (Gao et al., 2024)
- MOMENT (Goswami et al., 2024)
- OTIS (Turgut et al., 2026)
- TimesNet (Wu et al., 2023)
- Ti-MAE (Li et al., 2023)
- PatchTST (Nie et al., 2023)
- CroSSL (Deldari et al., 2024)
- TSLANet (Eldele et al., 2024)
- InvConvNet (Germain et al., 2025)
- Benchmarks:
- The authors only consider the 28 UEA multivariate archive (Bagnall et al., 2018) datasets while missing the commonly used 121 UCR (Dau et al., 2019) archive datasets
- The used ECG dataset only contains 5 subjects for testing. Authors should instead use larger datasets for benchmarking, like PTB-XL (Wagner et al., 2020) or MIMIC-IV-ECG (Gow et al., 2023)
- Most other benchmarks also contain only a few subjects (all less than 30). Authors should consider including benchmarks with more subjects, where available.
Overall, the evaluation setup is very limited and outdated, especially using an architecture without temporal interactions and lacking modern baselines, while using only benchmarks with few subjects. Therefore, the work cannot be judged in comparison to existing literature.
2. Limited Contribution and Motivation
The major contribution of the proposed work is the context-prediction task for pretraining. While the pretraining objective seems simple and effective within the authors’ evaluation setup, it relies on the properties of the InceptionTime architecture, which does not have temporal interactions. It is unclear how the method could generalize to models with temporal interactions. Therefore, the contribution may only be relevant within this extremely restricted evaluation scenario (without temporal interactions) and it is unclear why this scenario would be relevant.
While channel-independent pretraining seems interesting, the benefits are also not clearly motivated. Importantly, no channel interactions are possible during inference in this scenario. Furthermore, transformer-based time-series foundation models like OTIS (Turgut et al., 2026) already enable processing arbitrary channel dimensions, even without retraining the full model.
Overall, it is unclear what the proposed method contributes to the field, especially considering the large number of time-series foundation models and pretraining methods developed in the recent years. Authors would need to clearly motivate their work and evaluation setup considering recent progress and literature.
Questions for authors
- Why focus on the InceptionTime architecture?
4 comments
Thank you for your response. However, this does not resolve my concerns.
Model Architecture
CNN-based models are more and more being replaced by transformer-based models for time-series, as CNNs cannot capture long-range dependencies well. Considering that, I think transformer-based architectures need to be considered as well. With that in mind: does your representation learning approach also work with transformers and how well? Also, can it handle longer-range dependencies? Because anything longer than the patch-size is just handled via global-average pooling and in my opinion your proposed method relies in global-average pooling and may not work with other forms of long-range interactions.
Baselines
Even if these focus on architectural aspects, it would be interesting to compare their benchmark results with yours. These benchmark scores would set your results in context. While your method does not need to outperform those scores (because those foundation models may have been trained on way more data), I think it is important to see how far your experimental results are from SOTA results on the used benchmarks
Benchmarks
- UCR: I understand your argument for exclusion.
- ECG: if you want to demonstrate performance on ECG, you need reliable benchmarks. Testing on only 5 subjects is not reliable as they will not cover different patient distributions (e.g., age, sex, phenotypes/diseases) well. Even if it contains 16,645 patches/timepoints, these are highly correlated as they stem from the same few patients. Considering that there are larger datasets available for ECG, I do not understand why you chose this small benchmark.
Official Response by the Authors to Reviewer Z43V
Thank you for the prompt response and continued interest in our work.
Model architecture
We agree that transformers can model long-range dependencies and have proven effective in time series foundation models. In self-supervised representation learning for time series, however, the backbone is typically trained from scratch on each dataset, and transformers generally need large amounts of training data. As a result, they have seen little adoption in this setting.
Does your representation learning approach also work with transformers, and how well?
We did not adapt our approach to transformer backbones. We follow prior time series representation learning work (TNC, TS2Vec, TF-C, TS-TCC, SoftCLT, CaTT), which uses CNN encoders. TS2Vec's encoder ablation (Table 5) specifically shows a transformer backbone underperforming a CNN on the 128 UCR datasets.
Baselines
We agree that comparing with foundation models puts our results in context. Following your suggestion, we evaluated the open-source MOMENT-1-base model on the six large-scale datasets, using frozen embeddings and the same linear evaluation protocol as our other baselines. MOMENT-1-base model uses channel independence (CI) encoding, so we added C-Pred (CI) variant for comparison. We could not set up a reliable UniTS evaluation within the response period because its prompt tokens are dataset-specific.
| Model | ECG | HARTH | PAMAP2 | SKODA | SLEEP | WISDM2 | Avg. Acc. |
|---|---|---|---|---|---|---|---|
| *MOMENT-1-base (CI) | 78.11 | 93.35 | 73.82 | 99.24 | 76.14 | 68.05 | 81.45 |
| C-Pred | 76.15 | 93.43 | 70.54 | 99.08 | 85.15 | 63.87 | 81.37 |
| C-Pred (CI) | 79.88 | 94.45 | 80.95 | 99.62 | 85.16 | 66.09 | 84.36 |
Again, note that C-Pred and MOMENT represent different pretraining paradigms. C-Pred is a dataset-specific self-supervised learning (SSL) method, pretrained independently on the training split of each benchmark dataset without labels. In contrast, MOMENT is a time-series foundation model (FM), pretrained on a large and diverse corpus of time-series data.
*Edit: Improved and updated MOMENT results by resizing all input sequence lengths to 512 as recommended by the authors.
Benchmarks
Thank you for suggesting more diverse ECG datasets such as PTB-XL and MIMIC-IV-ECG. We will consider them in future work.
Thank you for your answer.
The additional foundation model results provide useful context.
However, the model architecture remains a critical issue. The field is moving to transformer-based architectures, even for smaller-scale scenarios. Therefore, the value of a new pretraining method that solely focuses on CNN-based models (and may not even be extendable to transformer-based models) seems unclear to me. Please explain the motivation why you focus only on CNN-based models and ignore transformer-based ones.
Regarding the ECG benchmarks: I understand that it may not be feasible to study your method on larger ECG benchmarks within the rebuttal period. However, it remains unclear why these have not been included so far as they are publicly available and commonly used datasets. And I repeat that testing on only 5 subjects CANNOT provide a robust analysis of ECG performance.
Official Response by the Authors to Reviewer Z43V
We appreciate the time you took to review our work and are glad you found the method compute-efficient and easy to deploy.
It is important to clarify that InceptionTime is a 1D convolutional network with multi-scale kernels (up to length 40) and stacked residual blocks. It models temporal interactions through its receptive field, and only the final aggregation is global average pooling. The same holds for the original dilated-CNN encoder of TS2Vec and the simple 1D CNN of CaTT.
We chose a single common backbone deliberately, so that differences reflect the pretext objective rather than the architecture. Table 3 also evaluates FCN and ResNet encoders.
Many of the listed methods are backbone architectures that focus on encoder design rather than on self-supervised learning (TimesNet, TSLANet, InvConvNet, OTIS), forecasting models (PatchTST), or foundation models pretrained on large external corpora (UniTS, MOMENT). These address a different setting from ours, which is a self-supervised pretext task on unlabeled data under a fixed compute budget.
We agree that channel independence (CI) cannot model cross-channel interactions at inference. This is consistent with our UEA results, where CI underperforms on datasets in which cross-variable dependencies matter, such as StandWalkJump.
As noted in our response to Reviewer BLfS, despite the lower average accuracy of the CI variant on UEA (70.67% vs. 72.05%), it achieves the best result on more individual datasets than channel mixing (8 wins vs. 4 wins). This suggests that explicit channel interactions are not needed to obtain good performance in several domains.
Our contribution is a lightweight, augmentation-free SSL objective for practitioners who pretrain directly on their own unlabeled data with limited compute. This complements, rather than competes with, foundation models.
Thank you for the detailed comments, and we hope our responses fully address your concerns.