- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The authors propose 3 contributions as ingredients for continual pretraining in TSFMs:
- Deep quantile regression supervision at intermediate layers, there is an additional unmentioned ingredient of trajectory regularization between intermediate layers.
- Progressive convergence schedule which is repeated restarts of the WSD schedule with decreased peak learning rate at each restart
- CauKerV2 an extension of CauKer a synthetic time-series generation framework with a wider synthetic generator family The authors claim that these ingredients enable continuable pretraining and transferable representations for downstream tasks such as time series classification and anomaly detection.
Critical strengths
- The authors perform detailed experiments with DQS to show that it improves forecasting performance and improves overall representational quality (Sec 4.2)
- The authors release the training and data pipeline
Critical weaknesses
- The ablations do not convincingly establish the benefit of the combined recipe. The unit ablations in table 1 do not sufficiently disentangle the effects of each of the pretraining tricks proposed in the paper. Based of the ablation experiments, practitioners can not draw the conclusion that all of these tricks yield a better overall pretraining recipe which is the central claim of the paper. Additionally,
- there almost no discussion or analysis of the trajectory regularization introduced in Sec 3.2 - it just adds another axis of complication to ablate.
- it unconvincing that CauKerV2 is an essential component of the pretraining recipe. KernelSynth makes up 30% of the pretraining data mix and CaukerV2 makes up 10%, yet the authors only ablate CauKerV2. They did not try ablating KernelSynth or even just ablating CauKerV2 with the original CauKer - there is no support for the authors proposed V2 of CauKer to stand in as a component for TSFM pretraining.
-
The TSFM baselines results in Figure 6 are not fair. Referencing the official gift-eval and TIME leaderboards on huggingface, the comparison numbers for the other TSFM baselines are misleading and unfair. The first concern is that Chronos-2-synth is used which is the Chronos-2 variant trained solely on synthetic data while Tabby is pretrained on synthetic and real data. It also appears that Chronos-2 beats Tabby on both metrics for TIME and on MASE for gift-eval. Second, the reported metrics for Toto 2.0 appear to be from the 22M parameter baseline while Tabby-pretrain has nearly 150M parameters. There are stronger Toto 2.0 baselines available (313M and 2.5B) which both outperform Tabby. Moreover, TimesFM 3.0 and PatchTST-FM-r2 were released before the submission deadline and should be added to the evaluation suite.
-
Details are missing for the downstream task experiments. For the classification and anomaly detection experiments, the procedure for extracting frozen representations for Tabby is described but unclear for the baselines. Also, it is unclear why the baselines for these experiments are so different from the baseline choice of the forecasting experiments.
-
PCS and CauKerV2 are oversold. PCS is simply just WSD-S with geometrically decaying learning rate. It is also well known that other TSFMs have used WSD schedules during pretraining (e.g. Toto 2.0). And as mentioned in Weakness 1, CauKerV2 is novel because it has a wider synthetic generator distribution than the original CauKer or KernelSynth, yet the authors never quantify this distributional enrichment or it's downstream effect on pretraining compared to other synthetic data generation methods beyond a unit ablation and a qualitative embedding map.
Questions for authors
-
(Regarding Weakness 1) Does the DQS ablation include ablating the trajectory regularization? What is the point of trajectory regularization - why is it included and how does it affect the downstream importance?
-
(Regarding Weakness 2) Is Chronos-2-synth the variant that is only trained on synthetic data? If so, why isn't Chronos-2 evaluated which would be a fair comparison given that Tabby is pretrained on synthetic and real data. What is the overall rationale for the choice of baselines in fig. 6? And could the authors make it clear that the Toto 2.0 baseline is the 22M checkpoint and similarly clarify the versions of the other baselines? Also, could the authors add TimesFM3.0, PatchTST-FM-r2, and the 313M Toto 2.0 models to the evaluation suite?
-
(Regarding Weakness 3) What is the rationale for the choice of baselines in the classification and anomaly detection experiments? Why cant the forecasters from fig. 6 be used as baselines for these downstream tasks? In table 3, TiRex is evaluated as well as in fig. 6, but why is Chronos evaluated and not Chronos-2 especially given the fact that Chronos is a substantially weaker autoregressive model with discrete tokenization? For table 4, Chronos is evaluated again and TimesFM instead of TimesFM 2.0 (or even TimesFM 3.0) - can the authors try to make the model selection more homogeneous between the forecasting and frozen representation evals?
-
(Regarding Weakness 4) Can you quantify the benefit of CauKerV2 over the original CauKer by performing the following ablations:
- replacing the 10% mix with CauKer
- replacing the 10% with KernelSynth so that 40% of the data mix is KernelSynth
- replacing the 30% KernelSynth mix with real data and fixing the CauKerV2 mix
10 comments
Component Ablations and Continuation
Regarding trajectory regularization and the DQS ablation (Weakness 1 and Question 1), the original “Remove DQS” ablation disables both intermediate quantile supervision and trajectory regularization. We now add separate continuation settings to examine their contributions more directly. All additional ablations below modify the 55K–110K continuation stage.
| Supervision during continuation | MASE ↓ | CRPS ↓ |
|---|---|---|
| Complete DQS | 0.7368 | 0.5038 |
| Intermediate supervision without trajectory regularization | 0.7428 | 0.5050 |
| Terminal supervision only | 0.7421 | 0.5054 |
Complete DQS reduces MASE by 0.81% and CRPS by 0.24% relative to intermediate supervision alone. This supports the incremental forecasting value of trajectory regularization.
Regarding CauKerV2 and KernelSynth (Weaknesses 1 and 4, and Question 4), we have added the requested source replacements, together with a comparison that preserves the ten primitive families and online sampling while removing SCM composition.
| Data intervention during continuation | MASE ↓ | CRPS ↓ |
|---|---|---|
| Complete recipe: 30% KernelSynth + 10% CauKerV2 | 0.7368 | 0.5038 |
| Replace 10% CauKerV2 with original CauKer | 0.7387 | 0.5042 |
| Remove SCM from CauKerV2 | 0.7413 | 0.5075 |
| Replace 10% CauKerV2 with real data | 0.7436 | 0.5059 |
| Replace 10% CauKerV2 with KernelSynth: 40% KernelSynth | 0.7503 | 0.5093 |
| Replace 30% KernelSynth with real data; retain 10% CauKerV2 | 0.7523 | 0.5117 |
The complete mixture achieves the lowest errors among these settings. Replacing CauKerV2 with additional KernelSynth degrades performance, while replacing KernelSynth with real data also degrades performance when CauKerV2 is retained. These results support complementary contributions from the two synthetic sources within the evaluated mixture.
Regarding the positioning of PCS (Weakness 4), we agree that WSD scheduling is established. PCS improve WSD/WSD-S by progressively attenuating restart peaks across continuation stages, motivated by the increasing sensitivity to aggressive restarts observed at later checkpoints.
We additionally compare alternative schedules over a complete continuation stage:
| Continuation schedule | MASE ↓ | CRPS ↓ |
|---|---|---|
| PCS | 0.7368 | 0.5038 |
| Constant learning rate of | 0.7422 | 0.5134 |
| Fixed-peak WSD | 0.7410 | 0.5029 |
| Cosine | 0.7384 | 0.5023 |
PCS obtains the lowest MASE, while fixed-peak WSD and Cosine achieve slightly lower CRPS.
We recognize that reviewer may approach these evaluation choices with different research contexts, and that our original presentation left too much of the rationale implicit. We will make the selection criteria, checkpoint versions, and evaluation protocols explicit. We hope the expanded comparisons clarify our intention to make the evaluation fair and reproducible.
Beyond PCS, DQS, and CauKerV2, we also release the data-processing code, complete pretraining code, and inference pipelines for forecasting, classification, and anomaly detection. These resources support open research and transparent comparisons within the TSFM community. We would be grateful if the reviewer could reconsider their assessment and score in light of the additional evidence, clarified evaluation, and community contribution.
Response to rebuttal comments
I very much appreciate the effort to add additional evaluations and ablations. I also especially appreciate the effort to make this work as open as possible. However, I still feel that some of my concerns are unaddressed and that the paper requires substantial revision.
Regarding weakness 1, the biggest concern I have is that there are no error bars are uncertainty quantifications over the evaluation data or tasks; and several existing TSFM works do [1,2,3]. I can be convinced that CauKer is a better synthetic data source than KernelSynth and even real data - however, the difference between the original data mix recipe and the one where CauKerV2 is replaced with the original CauKer seems minor and the significance of this difference is impossible to gauge without some sort of statistical uncertainty measure. Also, in the new ablation comparing terminal supervision vs intermediate supervision without trajectory regularization, the difference is similarly very small - doesnt this just suggest that 1) intermediate supervision doesnt do much and 2) trajectory regularization is the main driver of performance here? This ablation alone seems to undermine one of the main contributions established in the abstract.
Regarding weakness 2, the authors have adequately addressed my concerns and show that TABBY is a competitive forecaster.
Regarding weakness 3, the concerns are not completely addressed. Notably, the authors state that Chronos-2 mean pooled representations at layer 6 can achieve nearly the same classification accuracy as TABBY. I understand the instinct to evaluate all models according to the same representation extraction protocol for the random forest classifier - however, this does not necessarily make for a fair comparison. At evaluation time, there is no reason for a practitioner to limit themselves to the all-layers mean pool representation if layer 6 mean pooling yields better performance. Moreover, finding that layer 6 mean pooling in Chronos-2 yields good accuracy requires a basic sweep and can be applied to all the other TSFM baselines. The representation sensitivity basically establishes that TABBY can achieve the best accuracy under a single representation-extraction protocol.
Regarding weakness 4, I have concerns similar to those of weakness 1. PCS seems like quite a minor design decision and it remains unclear how it outperforms simple cosine decay from the new ablation table in the rebuttal other than ~0.02 difference in MASE without statistical quantification.
Overall, I feel that the central claims and results of the paper do not present a novel scientific contribution beyond introducing a pretraining recipe consisting of a few tricks that can yield a decent TSFM. For this reason I maintain my score. I do however highly value the "open research" paradigm and more so view this work similarly to the Marin open research effort (https://marin.community/) and feel that the authors should continue pushing along this front.
References
[1] https://openreview.net/pdf/3f1cd31c9960cda3f8a2c9f8ead1337fed4692b3.pdf
Baseline Selection and Evaluation Protocols
We understand the reviewer’s concerns about baseline selection. Our original choices were also motivated by fairness, particularly differences in pretraining data exposure and the availability of downstream evaluation protocols. We appreciate the opportunity to clarify them and broaden the comparisons.
Regarding forecasting baselines (Weakness 2 and Question 2), comparisons between pretrained TSFMs involve differences in training corpora, sampling policies, model sizes, and input capabilities. For example, Toto-2.0’s real data component consists of proprietary Datadog telemetry, preventing external researchers from reproducing identical training data conditions. Parameter count was therefore one consideration among several.
Chronos-2-Synth is indeed trained exclusively on synthetic data. We included it as a strong synthetic-only reference. We agree that the full Chronos-2 model provides an important additional comparison. Although it was cited and discussed throughout the paper, space constraints led to a compact selection in Figure 6. We have now expanded the evaluation to include Chronos-2, TimesFM-3, PatchTST-FM-r2, and Toto-2.0 313M/2.5B.
Regarding downstream baseline selection and representation extraction (Weakness 3 and Question 3), our original selection prioritized models with established evaluation pipelines on UCR and TSB-AD. This was motivated by the sensitivity of frozen representation evaluation to the extracted layer, token position, pooling method, and downstream estimator.
For example, in a separate controlled experiment on all 128 UCR datasets, Chronos-2’s mean-pooled representations achieve 79.79% accuracy at layer 6, compared the last context token representations with 71.90% at layer 12, a difference of 7.89 percentage points. This experiment demonstrat that representations choice alone can materially affect the result.
For another example, Toto-2 combines CPM with causal attention. Under its native attention pattern, a masked patch cannot directly use later observed patches, limiting its direct applicability to anomaly detection protocols that rely on bidirectional reconstruction. For classification, its patch level hidden states require an explicit readout to obtain a sequence level representation. Nevertheless, we agree that a more homogeneous model selection makes the comparison more informative.
We therefore extend the evaluation across all three tasks. Forecasting uses univariate inputs, matching Tabby’s channel-independent setting, while classification and anomaly detection evaluate frozen representations. These results follow our common evaluation protocol and may differ from leaderboard entries using model-specific configurations. The layer-sensitivity experiment above uses a separate diagnostic protocol.
The expanded comparison shows that stronger models outperform Tabby on some forecasting metrics, while Tabby remains competitive across tasks and achieves the highest classification accuracy among the evaluated models. We will revise the discussion accordingly.