- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces Saturn, a time series foundation model for classification and horizon-conditioned event forecasting, pretrained entirely on synthetic data. Its self-supervised objective combines joint-embedding prediction under a dual masking scheme, specifically an interior block and a terminal block sharing a single masking budget, with a contrastive loss over augmented views to discourage representation collapse. The synthetic corpus includes diverse physical dynamics, events with known precursors, precursor-free events, and event-free episodes. After pretraining, the encoder remains frozen and supports both tasks through lightweight task-specific heads. The event-forecasting head receives the prediction horizon and directly estimates the probability of an event at that horizon. The paper also introduces the Precursor Check, a protocol for assessing predictive signals before events while checking for temporal shortcuts and information leakage, and PrecursorBench, an event-forecasting benchmark comprising datasets that pass this check. Experiments show competitive classification performance on the UCR and UEA archives and event-forecasting performance comparable to substantially larger foundation models on the reported PrecursorBench corpora.
Critical strengths
- Event forecasting holds clear practical significance in applications such as equipment monitoring and fault early warning. Formulating it as a horizon-conditioned classification task provides a direct way to estimate event probabilities at specified future horizons.
- Experiments on UCR, UEA, and PrecursorBench provide evidence for the potential of transferring representations pretrained exclusively on synthetic data to real-world classification and event-forecasting tasks.
Critical weaknesses
- Font sizes in the table and figure captions appear to have been improperly modified from the standard template.
- Current ablation studies do not adequately disentangle the respective contributions of the prediction objective and the dual masking design. While removing the contrastive loss causes representation collapse, this observation does not reveal the actual gains of the JEPA prediction term over a purely contrastive approach. Furthermore, lacking comparisons with main-only and tail-only masking under an identical masking budget obscures the exact advantages of the dual masking scheme.
Questions for authors
- Would it be possible to provide a comparison between the full objective and a purely contrastive pretraining baseline, alongside an evaluation of the main-only, tail-only, and dual masking schemes?
- Could the authors clarify the inconsistencies regarding the masking settings between Table 4 and Appendix A.5, as well as the UCR accuracy discrepancies for Saturn-20M between Table 2 and Appendix H?
- Does selecting probe layers based on the average accuracy on the test set, as stated in Appendix H, introduce evaluation bias?
4 comments
Thank you for the detailed response and the additional experiments. The revision addresses most of my concerns.
After reading the other reviewers' questions and your responses, I have one follow-up question about the choice of pretraining corpus. Saturn performs comparably to the strongest baselines on classification and achieves the best event-forecasting score, with a modest margin over Mantis. Given these results, I would be interested in understanding how much the choice of pretraining data matters for each task. Two complementary comparisons could help clarify this:
- If one or more external baselines, such as Chronos-2, Mantis, or MantisV2, were pretrained from scratch on Saturn's synthetic corpus using their own objectives and comparable training budgets, how would they compare with Saturn on UCR-128, UEA-27, and the 15 real-world event corpora under the same evaluation protocol?
- Conversely, how would Saturn's results on these same benchmarks change if it were pretrained from scratch on an alternative corpus, such as the CauKer2M dataset used by MantisV2, while keeping its architecture, objective, and training budget fixed?
I appreciate the work you've put into the revision, and I'd be happy to raise my score if you can clarify how much Saturn's performance depends on its pretraining data.
We thank the reviewer for the follow-up and ran both comparisons. As the external model we used Mantis: it shares Saturn's tokeniser, so exchanging the objective and the corpus around it isolates the two factors. Every arm is pretrained from scratch on the same budget of 1.03M training windows, the size of the Saturn-dataset, and evaluated under the protocol of the paper (Appendix J): mean accuracy on UCR-128 and UEA-27, and horizon-averaged AUROC over the 15 real-world event corpora.
| Encoder | Pretraining corpus | UCR-128 | UEA-27 | Events (15 corpora) |
|---|---|---|---|---|
| Saturn-1M (0.9M), Saturn objective | Saturn-dataset | 0.7693 | 0.6998 | 0.7370 |
| Saturn-1M (0.9M), Saturn objective | CauKer | 0.7704 | 0.7077 | 0.7356 |
| Mantis (8.0M), Mantis objective | Saturn-dataset | 0.7979 | 0.7156 | 0.7260 |
| Mantis (8.0M), Mantis objective | CauKer | 0.8017 | 0.7293 | 0.7327 |
Mean of 3 pretraining seeds per arm. These are budget-matched runs, not the released checkpoints, so Table 2 is unchanged.
1. Saturn's performance depends mostly on the objective with a small help from pretraining data With architecture, objective and budget fixed, replacing CauKer with the Saturn-dataset raises Saturn's event score by +0.0014, higher in each of the three seeds, and lower in classification (−0.001 on UCR-128, −0.008 on UEA-27); Mantis on our data also has this minor drop in classification, and we agree that the corpus can be improved in future editions for classification. However, focussing on event-forecasting, Mantis does not benefit from our pre-training data: on the Saturn-dataset Mantis's event score drops by 0.007, and the likely reason is the objective. Mantis is trained only contrastively, while Saturn's objective also predicts the masked end of each window, a forecasting-like target that can use the event structure of the Saturn-dataset. The objective ablation of the general response shows that this masking carries Saturn's event gain (+0.0067 with dual masking over contrastive-only). Overall the ablations together show that both our pre-training data and our objective contribute to our winning scores on event-forecasting, while still being competitive on classification as well.
2. Whatever the corpus, Saturn leads Mantis on event forecasting. At the same training budget, Saturn-1M scores above Mantis on events on both corpora (+0.011 on the Saturn-dataset, +0.003 on CauKer), with about a ninth of Mantis's parameters. The lead grows with Saturn's size: in Table 2 it is +0.005 at Saturn-1M and +0.011 to +0.013 at Saturn-10M and Saturn-20M.
3. The classification gap at 1M parameters is likely a matter of model size. On either corpus, Mantis (8.0M) leads Saturn-1M (0.9M) by 0.029 to 0.031 on UCR-128 and 0.016 to 0.022 on UEA-27, but we could only retrain on the 1M in the interest of time. In Table 2 the gap closes as Saturn grows: Saturn-20M matches Mantis on UCR-128 (0.826 against 0.825) and surpasses it on UEA-27 (0.747 against 0.743). So, CauKer is slightly better for classification for both models, but the gap is likely less pronounced with Saturn-20M.
The Saturn-dataset adds a small gain on event forecasting that only Saturn's objective exploits, Saturn leads Mantis on events whatever the corpus, and the classification gap at small scale closes with model size.
Thank you for running the corpus comparisons. I hope you’ll include these results in the revised paper and extend the experiments to larger Saturn models. I’ve raised my score.
We thank the reviewer for seeing the practical point of the paper: estimating, at a given horizon, whether an event such as a fault is coming, and that synthetic pretraining can carry this to real data. The question about choosing probe layers on test accuracy was the right one to ask, and it led to one of the central changes in this revision. Prompted by this review and related points from the other reviewers, we improved the evaluation protocol and re-ran every model under it (see the official comment to all authors). Every model is now read through the same readouts and heads, with the readout chosen on validation data only. New results are in the revised PDF, and the main results are reported in the table below:
On events, Saturn leads Chronos-2 by +0.024, far larger than the pretraining-seed s.d. (0.0003 to 0.0015 across the three sizes) and the head-seed s.d. (0.0007). On classification it is on par.
We also thank the reviewer for the two points on presentation and ablations:
We now turn to the reviewer's questions:
Would it be possible to provide a comparison between the full objective and a purely contrastive pretraining baseline, alongside an evaluation of the main-only, tail-only, and dual masking schemes?
Could the authors clarify the inconsistencies regarding the masking settings between Table 4 and Appendix A.5, as well as the UCR accuracy discrepancies for Saturn-20M between Table 2 and Appendix H?
Does selecting probe layers based on the average accuracy on the test set, as stated in Appendix H, introduce evaluation bias?