acceptodds
Under review as a conference paper at ICLR 2027

Tabby-Pretrain: A Recipe for Continuable Pretraining of Time Series Foundation Models

Abstract

Pretraining a time series foundation model is costly, making it desirable for a training run to remain effective as it is extended. In this paper, we develop a pretraining recipe that supports continuable forecasting improvements across successive pretraining stages through the coordinated design of optimization, supervision, and data, and instantiate it in Tabby-Pretrain, an encoder-only Transformer-based TSFM. The resulting recipe combines a progressive convergence schedule (PCS), which repeatedly produces annealed checkpoints; deep quantile supervision (DQS), which directly supervises intermediate layers; and CauKerV2, an online generator that composes diverse temporal dynamics through randomly sampled structural causal models. Controlled experiments with the 40M-parameter Tabby-Small show that replacing PCS, removing DQS, or replacing CauKerV2 with additional real data degrades forecasting at the same training budget. We then apply the recipe to a 146M-parameter model. Its reported training trajectory improves across three stable–decay stages, and the final pretrained backbone is competitive in forecasting while supporting frozen representation transfer to classification and anomaly detection.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Foundation Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2jn5OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The authors propose 3 contributions as ingredients for continual pretraining in TSFMs:

  1. Deep quantile regression supervision at intermediate layers, there is an additional unmentioned ingredient of trajectory regularization between intermediate layers.
  2. Progressive convergence schedule which is repeated restarts of the WSD schedule with decreased peak learning rate at each restart
  3. CauKerV2 an extension of CauKer a synthetic time-series generation framework with a wider synthetic generator family The authors claim that these ingredients enable continuable pretraining and transferable representations for downstream tasks such as time series classification and anomaly detection.
Critical strengths
  • The authors perform detailed experiments with DQS to show that it improves forecasting performance and improves overall representational quality (Sec 4.2)
  • The authors release the training and data pipeline
Critical weaknesses
  1. The ablations do not convincingly establish the benefit of the combined recipe. The unit ablations in table 1 do not sufficiently disentangle the effects of each of the pretraining tricks proposed in the paper. Based of the ablation experiments, practitioners can not draw the conclusion that all of these tricks yield a better overall pretraining recipe which is the central claim of the paper. Additionally,
  • there almost no discussion or analysis of the trajectory regularization introduced in Sec 3.2 - it just adds another axis of complication to ablate.
  • it unconvincing that CauKerV2 is an essential component of the pretraining recipe. KernelSynth makes up 30% of the pretraining data mix and CaukerV2 makes up 10%, yet the authors only ablate CauKerV2. They did not try ablating KernelSynth or even just ablating CauKerV2 with the original CauKer - there is no support for the authors proposed V2 of CauKer to stand in as a component for TSFM pretraining.
  1. The TSFM baselines results in Figure 6 are not fair. Referencing the official gift-eval and TIME leaderboards on huggingface, the comparison numbers for the other TSFM baselines are misleading and unfair. The first concern is that Chronos-2-synth is used which is the Chronos-2 variant trained solely on synthetic data while Tabby is pretrained on synthetic and real data. It also appears that Chronos-2 beats Tabby on both metrics for TIME and on MASE for gift-eval. Second, the reported metrics for Toto 2.0 appear to be from the 22M parameter baseline while Tabby-pretrain has nearly 150M parameters. There are stronger Toto 2.0 baselines available (313M and 2.5B) which both outperform Tabby. Moreover, TimesFM 3.0 and PatchTST-FM-r2 were released before the submission deadline and should be added to the evaluation suite.

  2. Details are missing for the downstream task experiments. For the classification and anomaly detection experiments, the procedure for extracting frozen representations for Tabby is described but unclear for the baselines. Also, it is unclear why the baselines for these experiments are so different from the baseline choice of the forecasting experiments.

  3. PCS and CauKerV2 are oversold. PCS is simply just WSD-S with geometrically decaying learning rate. It is also well known that other TSFMs have used WSD schedules during pretraining (e.g. Toto 2.0). And as mentioned in Weakness 1, CauKerV2 is novel because it has a wider synthetic generator distribution than the original CauKer or KernelSynth, yet the authors never quantify this distributional enrichment or it's downstream effect on pretraining compared to other synthetic data generation methods beyond a unit ablation and a qualitative embedding map.

Questions for authors
  1. (Regarding Weakness 1) Does the DQS ablation include ablating the trajectory regularization? What is the point of trajectory regularization - why is it included and how does it affect the downstream importance?

  2. (Regarding Weakness 2) Is Chronos-2-synth the variant that is only trained on synthetic data? If so, why isn't Chronos-2 evaluated which would be a fair comparison given that Tabby is pretrained on synthetic and real data. What is the overall rationale for the choice of baselines in fig. 6? And could the authors make it clear that the Toto 2.0 baseline is the 22M checkpoint and similarly clarify the versions of the other baselines? Also, could the authors add TimesFM3.0, PatchTST-FM-r2, and the 313M Toto 2.0 models to the evaluation suite?

  3. (Regarding Weakness 3) What is the rationale for the choice of baselines in the classification and anomaly detection experiments? Why cant the forecasters from fig. 6 be used as baselines for these downstream tasks? In table 3, TiRex is evaluated as well as in fig. 6, but why is Chronos evaluated and not Chronos-2 especially given the fact that Chronos is a substantially weaker autoregressive model with discrete tokenization? For table 4, Chronos is evaluated again and TimesFM instead of TimesFM 2.0 (or even TimesFM 3.0) - can the authors try to make the model selection more homogeneous between the forecasting and frozen representation evals?

  4. (Regarding Weakness 4) Can you quantify the benefit of CauKerV2 over the original CauKer by performing the following ablations:

  • replacing the 10% mix with CauKer
  • replacing the 10% with KernelSynth so that 40% of the data mix is KernelSynth
  • replacing the 30% KernelSynth mix with real data and fixing the CauKerV2 mix
10 comments
AuthorsOpenReview ↗

Baseline Selection and Evaluation Protocols

We understand the reviewer’s concerns about baseline selection. Our original choices were also motivated by fairness, particularly differences in pretraining data exposure and the availability of downstream evaluation protocols. We appreciate the opportunity to clarify them and broaden the comparisons.

Regarding forecasting baselines (Weakness 2 and Question 2), comparisons between pretrained TSFMs involve differences in training corpora, sampling policies, model sizes, and input capabilities. For example, Toto-2.0’s real data component consists of proprietary Datadog telemetry, preventing external researchers from reproducing identical training data conditions. Parameter count was therefore one consideration among several.

Chronos-2-Synth is indeed trained exclusively on synthetic data. We included it as a strong synthetic-only reference. We agree that the full Chronos-2 model provides an important additional comparison. Although it was cited and discussed throughout the paper, space constraints led to a compact selection in Figure 6. We have now expanded the evaluation to include Chronos-2, TimesFM-3, PatchTST-FM-r2, and Toto-2.0 313M/2.5B.

Regarding downstream baseline selection and representation extraction (Weakness 3 and Question 3), our original selection prioritized models with established evaluation pipelines on UCR and TSB-AD. This was motivated by the sensitivity of frozen representation evaluation to the extracted layer, token position, pooling method, and downstream estimator.

For example, in a separate controlled experiment on all 128 UCR datasets, Chronos-2’s mean-pooled representations achieve 79.79% accuracy at layer 6, compared the last context token representations with 71.90% at layer 12, a difference of 7.89 percentage points. This experiment demonstrat that representations choice alone can materially affect the result.

For another example, Toto-2 combines CPM with causal attention. Under its native attention pattern, a masked patch cannot directly use later observed patches, limiting its direct applicability to anomaly detection protocols that rely on bidirectional reconstruction. For classification, its patch level hidden states require an explicit readout to obtain a sequence level representation. Nevertheless, we agree that a more homogeneous model selection makes the comparison more informative.

We therefore extend the evaluation across all three tasks. Forecasting uses univariate inputs, matching Tabby’s channel-independent setting, while classification and anomaly detection evaluate frozen representations. These results follow our common evaluation protocol and may differ from leaderboard entries using model-specific configurations. The layer-sensitivity experiment above uses a separate diagnostic protocol.

ModelGIFT MASE ↓GIFT CRPS ↓TIME MASE ↓TIME CRPS ↓UCR Acc. (%) ↑TSB-AD-U VUS-PR ↑
Tabby-Pretrain0.70400.48270.67790.566479.990.4300
Chronos-20.70830.49690.66380.557671.900.3617
TimesFM-30.69530.47730.64820.544570.280.1918
PatchTST-FM-r20.70320.48370.65440.550676.870.4230
Toto-2.0 313M0.70300.48370.64610.543668.540.2340
Toto-2.0 2.5B0.69670.47920.64380.542068.500.2212
TiRex0.72360.49850.68310.573174.570.2919
TimesFM-20.75860.54850.71760.620169.670.3029
PatchTST-FM-r10.71720.49070.67400.567776.810.3982
Moirai-20.72810.51640.73130.605474.410.3148
Chronos-2-Synth0.73660.51010.69060.580271.830.3547
Chronos-Bolt0.80960.56220.73270.619876.060.4467
Sundial0.75640.56020.72190.658476.300.3187
Timer-S10.70430.49880.67520.576077.640.3436

The expanded comparison shows that stronger models outperform Tabby on some forecasting metrics, while Tabby remains competitive across tasks and achieves the highest classification accuracy among the evaluated models. We will revise the discussion accordingly.

AuthorsOpenReview ↗

Component Ablations and Continuation

Regarding trajectory regularization and the DQS ablation (Weakness 1 and Question 1), the original “Remove DQS” ablation disables both intermediate quantile supervision and trajectory regularization. We now add separate continuation settings to examine their contributions more directly. All additional ablations below modify the 55K–110K continuation stage.

Supervision during continuationMASE ↓CRPS ↓
Complete DQS0.73680.5038
Intermediate supervision without trajectory regularization0.74280.5050
Terminal supervision only0.74210.5054

Complete DQS reduces MASE by 0.81% and CRPS by 0.24% relative to intermediate supervision alone. This supports the incremental forecasting value of trajectory regularization.

Regarding CauKerV2 and KernelSynth (Weaknesses 1 and 4, and Question 4), we have added the requested source replacements, together with a comparison that preserves the ten primitive families and online sampling while removing SCM composition.

Data intervention during continuationMASE ↓CRPS ↓
Complete recipe: 30% KernelSynth + 10% CauKerV20.73680.5038
Replace 10% CauKerV2 with original CauKer0.73870.5042
Remove SCM from CauKerV20.74130.5075
Replace 10% CauKerV2 with real data0.74360.5059
Replace 10% CauKerV2 with KernelSynth: 40% KernelSynth0.75030.5093
Replace 30% KernelSynth with real data; retain 10% CauKerV20.75230.5117

The complete mixture achieves the lowest errors among these settings. Replacing CauKerV2 with additional KernelSynth degrades performance, while replacing KernelSynth with real data also degrades performance when CauKerV2 is retained. These results support complementary contributions from the two synthetic sources within the evaluated mixture.

Regarding the positioning of PCS (Weakness 4), we agree that WSD scheduling is established. PCS improve WSD/WSD-S by progressively attenuating restart peaks across continuation stages, motivated by the increasing sensitivity to aggressive restarts observed at later checkpoints.

We additionally compare alternative schedules over a complete continuation stage:

Continuation scheduleMASE ↓CRPS ↓
PCS0.73680.5038
Constant learning rate of 10−510^{-5}0.74220.5134
Fixed-peak WSD0.74100.5029
Cosine0.73840.5023

PCS obtains the lowest MASE, while fixed-peak WSD and Cosine achieve slightly lower CRPS.

We recognize that reviewer may approach these evaluation choices with different research contexts, and that our original presentation left too much of the rationale implicit. We will make the selection criteria, checkpoint versions, and evaluation protocols explicit. We hope the expanded comparisons clarify our intention to make the evaluation fair and reproducible.

Beyond PCS, DQS, and CauKerV2, we also release the data-processing code, complete pretraining code, and inference pipelines for forecasting, classification, and anomaly detection. These resources support open research and transparent comparisons within the TSFM community. We would be grateful if the reviewer could reconsider their assessment and score in light of the additional evidence, clarified evaluation, and community contribution.

Reviewer_2jn5OpenReview ↗

Response to rebuttal comments

I very much appreciate the effort to add additional evaluations and ablations. I also especially appreciate the effort to make this work as open as possible. However, I still feel that some of my concerns are unaddressed and that the paper requires substantial revision.

Regarding weakness 1, the biggest concern I have is that there are no error bars are uncertainty quantifications over the evaluation data or tasks; and several existing TSFM works do [1,2,3]. I can be convinced that CauKer is a better synthetic data source than KernelSynth and even real data - however, the difference between the original data mix recipe and the one where CauKerV2 is replaced with the original CauKer seems minor and the significance of this difference is impossible to gauge without some sort of statistical uncertainty measure. Also, in the new ablation comparing terminal supervision vs intermediate supervision without trajectory regularization, the difference is similarly very small - doesnt this just suggest that 1) intermediate supervision doesnt do much and 2) trajectory regularization is the main driver of performance here? This ablation alone seems to undermine one of the main contributions established in the abstract.

Regarding weakness 2, the authors have adequately addressed my concerns and show that TABBY is a competitive forecaster.

Regarding weakness 3, the concerns are not completely addressed. Notably, the authors state that Chronos-2 mean pooled representations at layer 6 can achieve nearly the same classification accuracy as TABBY. I understand the instinct to evaluate all models according to the same representation extraction protocol for the random forest classifier - however, this does not necessarily make for a fair comparison. At evaluation time, there is no reason for a practitioner to limit themselves to the all-layers mean pool representation if layer 6 mean pooling yields better performance. Moreover, finding that layer 6 mean pooling in Chronos-2 yields good accuracy requires a basic sweep and can be applied to all the other TSFM baselines. The representation sensitivity basically establishes that TABBY can achieve the best accuracy under a single representation-extraction protocol.

Regarding weakness 4, I have concerns similar to those of weakness 1. PCS seems like quite a minor design decision and it remains unclear how it outperforms simple cosine decay from the new ablation table in the rebuttal other than ~0.02 difference in MASE without statistical quantification.

Overall, I feel that the central claims and results of the paper do not present a novel scientific contribution beyond introducing a pretraining recipe consisting of a few tricks that can yield a decent TSFM. For this reason I maintain my score. I do however highly value the "open research" paradigm and more so view this work similarly to the Marin open research effort (https://marin.community/) and feel that the authors should continue pushing along this front.

References

[1] https://openreview.net/pdf/3f1cd31c9960cda3f8a2c9f8ead1337fed4692b3.pdf

[2] https://arxiv.org/abs/2610.41956v1

[3] https://arxiv.org/abs/2602.06909

Reviewer_bQmpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a recipe for continuable pretraining of time-series foundation models, combining three components: a progressive convergence schedule (PCS), which repeats stable–decay stages with decreasing peak learning rates; deep quantile supervision (DQS), which combines intermediate-layer quantile supervision with prediction-trajectory regularization; and CauKerV2, an online synthetic-data generator that composes diverse primitive processes through structural causal models. The authors conduct component ablations using the approximately 40M-parameter TABBY-SMALL and demonstrate three-stage pretraining of an approximately 146M-parameter model. The resulting model achieves competitive forecasting performance on GIFT-Eval and TIME, while its frozen representations also support classification and anomaly detection.

Critical strengths
  1. The complete model is competitive and demonstrates utility across multiple tasks. The approximately 146M-parameter model achieves MASE/CRPS of 0.7040/0.4827 on GIFT-Eval and 0.6779/0.5664 on TIME. Its frozen encoder also supports UCR classification and TSB-AD-U anomaly detection. These results indicate that the recipe learns useful time-series representations and has practical value.
  2. The diagnostic experiments are carefully constructed, and the authors acknowledge their evidential limits. The one-step DQS experiment controls optimizer state, raw gradient norm, and actual parameter-displacement magnitude. The restart analysis evaluates multiple checkpoints and candidate learning rates. The authors also acknowledge that local probes cannot replace full-stage continuation experiments and that validation-sample bootstrapping does not capture variability across independent training runs. These controls and qualifications help readers interpret the findings accurately.
Critical weaknesses
  1. PCS has supporting evidence under a fixed training budget, but its specific advantage as a continuation strategy remains insufficiently validated. Table 1 shows that PCS outperforms the evaluated Cosine and WSD-S variants on TABBY-SMALL at 110K steps. This is meaningful performance evidence. However, the comparison primarily reports final outcomes of complete training runs and does not isolate the contribution of early training decisions from that of continuation after an annealed checkpoint. Figure 5 shows progressively better stage-end performance at 55K, 110K, and 165K steps, but the model also receives additional training. This trajectory alone cannot establish how much improvement comes specifically from the PCS restart strategy.Appendix H supports the observation that later checkpoints are more sensitive to aggressive restarts, but it does not directly establish the long-term benefit of progressively decreasing restart peaks. For every tested checkpoint, the best candidate in the 1,500-step is the constant learning rate of (10^{-5}). This demonstrates a short-term penalty for higher restart rates, rather than showing that the optimal restart peak decreases across stages or that halving the peak is preferable to alternative continuation schedules.I appreciate the authors’ explicit acknowledgment of this limitation, and I do not interpret the short-horizon results as disproving the long-term value of PCS. The decisive missing comparison is whether, starting from the same stage-end checkpoint and optimizer state, PCS outperforms constant-low-learning-rate continuation and other reasonable schedules over a complete, matched continuation budget. Without this comparison, the evidence for the paper’s central continuation strategy remains incomplete.
  2. The independent contributions and cross-size applicability of the components are insufficiently established, while benefit stability and computational overhead remain underreported. First, the ablations do not sufficiently separate the constituent designs. DQS combines intermediate quantile supervision with trajectory regularization, but Table 1 removes DQS as a whole rather than comparing intermediate supervision alone against the complete objective. Appendix G explicitly disables trajectory regularization, so it cannot establish the additional value of the linear prediction-trajectory constraint. Similarly, replacing CauKerV2’s 10% sampling share with real data shows that this synthetic source is useful within the current mixture, but does not separate the contributions of the primitive generators, SCM composition, and online sampling. A comparison against the original CauKer would assess the overall improvement of V2, while a comparison retaining the primitive generators but removing SCM composition would more directly assess the contribution of the SCM.Second, it is unclear whether the component-level findings transfer across model sizes. Controlled ablations are primarily conducted on the approximately 40M-parameter model, whereas the approximately 146M-parameter model also changes the data mixture, supervised exits, and scheduling configuration. Strong performance of the larger model demonstrates that the recipe is usable in another configuration, but does not establish that individual component gains persist across sizes. This is particularly relevant to DQS: supervised-exit placement, exit count, and auxiliary-loss weights may need to change with depth. Because the auxiliary losses are summed over exits, changing the number of exits may also change their strength relative to the terminal objective. The paper does not sufficiently explain the transfer rules for these settings. Appendix H provides useful evidence of similar short-term restart-sensitivity trends at two sizes, but does not replace cross-size component ablations.Finally, Table 1 does not report variability across independent pretraining runs, and the bootstrap analyses cannot substitute for this information. DQS requires additional decoding, loss computation, and backpropagation, so equal training steps do not imply equal computational cost. Given the relatively modest component gains, repeated results for key comparisons and measured training throughput would materially affect the assessment of reliability and practical value.
Questions for authors
  1. Full-stage continuation comparison: Do you have results comparing PCS, constant-(10^{-5}) continuation, and an explicitly specified WSD-S continuation schedule from the same 55K checkpoint and optimizer state, with matched additional training budgets and data conditions? A targeted comparison on TABBY-SMALL would already be informative. Please report both forecasting performance and training cost.
  2. Incremental value of trajectory regularization: How does intermediate quantile supervision alone compare with the complete DQS objective after full training? Is there direct evidence that the linear interpolation constraint provides additional predictive gains or improves training stability?
  3. Source of CauKerV2’s gains: Have you compared CauKerV2 against the original CauKer under matched settings to assess the overall improvement? Do you have a comparison that preserves the primitive generators and online sampling while removing SCM composition, thereby isolating its contribution?
  4. Applicability across model sizes: Which recipe settings were retained when moving from TABBY-SMALL to the full model, which were changed, and on what basis? In particular, how are supervised exits and auxiliary-loss weights adjusted with model depth? Are there key component ablations at the two existing sizes showing that the gains persist?
  5. Stability and overhead: How many independent training seeds were used for each setting in Table 1? Can you report repeated results for key comparisons and training throughput or total GPU-hours with and without DQS? These questions concern validation of the current contributions. The first most directly affects my assessment of the central claim. Cross-size validation could prioritize the two existing model sizes rather than requiring additional, larger models.
1 comment

We thank the reviewer for the careful assessment and for recognizing the model’s practical value and the controls in our diagnostic experiments. Your comments help identify where more direct comparisons and clearer reporting are needed.

Regarding full-stage continuation (Weakness 1 and Question 1), we add experiments that continue Tabby-Small from the 55K checkpoint for another 55K steps using alternative schedules. Each ablation branch restores the same 55K model parameters and full AdamW optimizer state. All results below are on GIFT-Eval, with lower values indicating better performance.

Continuation scheduleMASE ↓CRPS ↓
PCS (Table 1 reference)0.73680.5038
Constant learning rate of 10−510^{-5}0.74220.5134
Fixed-peak WSD0.74100.5029
Cosine0.73840.5023

PCS achieves the lowest MASE and improves both metrics over constant low learning rate continuation. This comparison extends the evidence to one complete continuation stage.

Regarding the incremental value of trajectory regularization (Weakness 2 and Question 2), we add an intermediate-supervision-only setting and a terminal-supervision-only setting to separate the two auxiliary terms during continuation.

Supervision during continuationMASE ↓CRPS ↓
Complete DQS (Table 1 reference)0.73680.5038
Intermediate supervision without trajectory regularization0.74280.5050
Terminal supervision only0.74210.5054

Complete DQS reduces MASE by 0.81% and CRPS by 0.24% relative to intermediate supervision alone. This provides predictive evidence for the additional value of trajectory regularization. Intermediate supervision alone gives mixed results relative to terminal-only supervision, so we will distinguish the benefit of the complete objective from the evidence for each term individually.

Regarding the source of CauKerV2’s gains (Weakness 2 and Question 3), we add comparisons against the original CauKer and against direct sampling from the same ten primitive families without SCM composition.

Synthetic source during continuationMASE ↓CRPS ↓
CauKerV2 (Table 1 reference)0.73680.5038
Original CauKer0.73870.5042
Same ten generator families without SCM0.74130.5075

CauKerV2 obtains lower errors than the original CauKer, although the differences are modest. Removing SCM while retaining the primitive families and online sampling increases both errors. Relative to direct primitive sampling, complete CauKerV2 reduces MASE by 0.61% and CRPS by 0.73%.

Regarding applicability across model sizes (Weakness 2 and Question 4), Using smaller scale experiments to guide larger scale training has precedents in T5, BERT, and Chinchilla. We respect and understand the reviewer's concerns. To demonstrate it, we further provide a direct cross-size check: DQS reduces GIFT-Eval MASE by 1.44% on Tabby-Small and 3.82% on Tabby-Pretrain, relative to their respective counterparts without DQS. Both models retain three uniformly spaced intermediate supervision exits, keeping their number fixed as depth increases from 12 to 20 layers. These results support DQS’s applicability across both the 40M and 146M scales.

Regarding independent runs and computational overhead (Weakness 2 and Question 5), given the substantial pretraining cost, Table 1 uses one training seed per setting. This follows the practice of Google's TimesFM and ICML Oral Sundial. Our continuation and ablation results provide consistent evidence of the recipe’s benefits. At fixed model size, batch size, and input length, DQS increases the computational cost per training step—measured in FLOPs and including the forward pass, backward pass, and optimizer update—by 5.94%. The mean training step time is approximately 375.2 ms without DQS and 442.5 ms with complete DQS.

We appreciate the reviewer’s emphasis on direct experimental validation. In response, we have added full-stage continuation comparisons, separate DQS ablations, and comparisons with the original CauKer and direct primitive sampling without SCM. The additional cross-size evaluation shows that DQS reduces GIFT-Eval MASE by 1.44% at 40M parameters and 3.82% at 146M parameters, supporting its applicability at both scales. These results strengthen the empirical basis of the recipe, while our released pretraining code and data generation pipeline allow the community to reproduce and extend it. We hope these additions address the main concerns underlying the original assessment, and would be grateful if the reviewer could reconsider their evaluation and score.

Reviewer_nnsSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Tabby-Pretrain, a training recipe designed to support continued pretraining of time series foundation models. It combines three components: a progressive convergence schedule (PCS) for learning rate scheduling, deep quantile supervision (DQS) for training intermediate layers, and CauKerV2 for generating diverse synthetic time series. The authors evaluate the recipe through ablation studies and train a 146M-parameter model, which achieves competitive results on zero-shot forecasting benchmarks. They also show that the pretrained model can be used for classification and anomaly detection without fine-tuning the encoder.

Critical strengths
  1. The paper focuses on continuable pretraining, which is a meaningful problem given the high computational cost of training time series foundation models. I think this is a useful direction to explore, especially as larger models become increasingly expensive to train.
  2. The authors evaluate their model on several benchmarks covering forecasting, classification, and anomaly detection, with competitive results across these tasks. They also include component ablations and additional analyses of intermediate layers and learning rate restarts, which help explain how the proposed recipe works.
Critical weaknesses
  1. The three main components, PCS, DQS, and CauKerV2, are largely built on existing ideas, including learning rate scheduling, deep supervision, and synthetic data generation. While combining them into a pretraining recipe is useful, I find the methodological novelty somewhat limited, as it is not clear what fundamentally new insights this work offers beyond existing approaches.
  2. Continuable pretraining is the main focus of this paper, but the experimental evidence is not fully convincing. Figure 5 shows that the model continues to improve across three PCS stages, but there is no direct comparison with other learning rate schedules over the same continuation stages, making it hard to tell whether PCS actually provides a clear advantage for continued pretraining.
Questions for authors
  1. The paper focuses on continuable pretraining, but Figure 5 only shows how the model performs across three stages using PCS. Although Table 1 compares PCS with other learning rate schedules at a fixed training budget, it is still unclear whether PCS is actually better for continued pretraining. Could the authors compare PCS with Cosine or WSD-S across the same training stages to better support this claim?
  2. DQS includes both intermediate quantile supervision and prediction trajectory regularization, but Table 1 only reports the results of removing DQS entirely. Could the authors provide separate ablations for these two components?
  3. CauKerV2 combines different synthetic time series generators using structural causal models, but it is not clear whether the SCM-based composition is actually necessary. Could the authors compare it with a simpler approach that samples directly from the same ten generators without SCM composition, while keeping the training budget unchanged?
1 comment

We thank the reviewer for the thoughtful feedback and for recognizing the value of continuable pretraining.

Regarding novelty (Weakness 1), our contribution lies in specific advances in optimization, supervision, and data preparation for efficient TSFM pretraining. DQS is, to our knowledge, the first supervision method explicitly targeting functional redundancy in TSFMs. PCS progressively lowers peak learning rates across stages, motivated by the increasing sensitivity to aggressive restarts as training progresses. CauKerV2 introduces additional generator families to broaden temporal diversity. Each component therefore extends existing ideas to address a concrete pretraining challenge. The central insight is that effective continuation depends on the model’s evolving optimization sensitivity and utilization of depth.

Regarding the continuation advantage of PCS (Weakness 2 and Question 1), we understand that the reviewer worry the trajectory in Figure 5 alone cannot distinguish the effect of PCS from that of additional training. We therefore add experiments that continue Tabby-Small from the 55K checkpoint for another 55K steps using alternative schedules. All results below are on GIFT-Eval, with lower values indicating better performance.

Continuation scheduleMASE ↓CRPS ↓
PCS (Table 1 reference)0.73680.5038
Constant low learning rate0.74220.5134
Fixed-peak WSD0.74100.5029
Cosine0.73840.5023

PCS achieves the lowest MASE among these settings. These results provide additional evidence for PCS as an effective continuation schedule. We will make this distinction explicit and clarify that this comparison covers one complete continuation stage on Tabby-Small, complementing the three-stage trajectory of the full model.

Regarding the two components of DQS (Question 2), we add a setting that retains intermediate quantile supervision while disabling trajectory regularization, together with a terminal-supervision-only setting.

Supervision during continuationMASE ↓CRPS ↓
Complete DQS (Table 1 reference)0.73680.5038
Intermediate supervision without trajectory regularization0.74280.5050
Terminal supervision only0.74210.5054

Compared with intermediate supervision alone, complete DQS reduces MASE by 0.81% and CRPS by 0.24%. This supports the incremental value of trajectory regularization when intermediate supervision is present. Intermediate supervision alone gives mixed results relative to terminal-only supervision, so we will distinguish the benefit of the complete objective from the evidence for each term individually.

Regarding SCM composition in CauKerV2 (Question 3), we add a comparison that samples directly from the same ten primitive generator families without SCM composition, while retaining online generation, the synthetic data sampling share, and the continuation budget.

Synthetic generationMASE ↓CRPS ↓
CauKerV2 with SCM (Table 1 reference)0.73680.5038
Same ten generator families without SCM0.74130.5075

Removing SCM increases both errors. Relative to direct primitive sampling, the complete generator reduces MASE by 0.61% and CRPS by 0.73%. This supports an additional contribution from SCM composition.

Beyond our methodological contributions in PCS, DQS, and CauKerV2, in the supplementary material we have also released the complete pretraining code and data generation pipeline, enabling the community to reproduce, extend, and build upon our work. We hope that the additional experiments help address the reviewer’s concerns. We would be grateful if the reviewer could reconsider their assessment and score in light of this new evidence and our contribution to open and reproducible TSFM research.

Comments on the paper

1 comment

We thank the reviewers for their careful assessments and constructive discussion. Their feedback has helped us strengthen the evaluation and clarify the scope of the contributions.

  • Expanded evaluation. We added Chronos-2, TimesFM-3, PatchTST-FM-r2, and larger Toto-2.0 checkpoints under a common univariate forecasting setting, alongside additional classification and anomaly detection comparisons using frozen representations.

  • Additional controlled experiments. We added full stage continuation comparisons from the same 55K checkpoint and AdamW state, further DQS loss ablations, and synthetic source comparisons covering the original CauKer, SCM removal, KernelSynth, and real data replacements.

  • Clearer claims. We now consistently define DQS as a joint objective combining intermediate quantile supervision and trajectory regularization, and position PCS as a WSD-based continuation policy.

The revised results support Tabby’s competitive forecasting performance and the utility of its frozen representations, while making the scope of each experimental finding more explicit. We release data generators, complete pretraining and evaluation pipelines to support reproducible research. We appreciate the reviewers’ time and consideration of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.