- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies data selection for pretraining time-series foundation models. The proposed method uses a frozen reference model to score candidate forecasting windows by their loss and retains the intermediate-loss windows within each source dataset. Pretraining experiments across Chronos-Bolt, Moirai, and TimesFM show that retaining 80% of the windows can outperform both random selection and full-data training under the same compute budget. Additional analyses examine the impact of the selected loss region, dataset stratification, and transferring scores across reference model sizes and architectures.
Critical strengths
-
The paper tackles a useful practical question with a simple, intuitive heuristic that can easily plug into existing pretraining pipelines. A nice practical advantage is that for a fixed candidate pool and reference model, the offline scoring only needs to be done once, and the cached scores can be reused across different retention ratios and compatible target models. Pretraining three distinct model families from scratch also gives solid evidence that the approach works across different target architectures.
-
The experimental setup is well-structured, comparing against full-data training and size-matched random baselines while thoroughly ablating the loss region and dataset stratification components. I also really appreciate that the authors transparently report negative results—such as showing that using TimesFM as a reference model fails to improve Moirai's pretraining performance.
Critical weaknesses
-
I think it is still unclear how reliable the reported improvements are. At 80% retention, aggregate MASE and CRPS improve by about 0.7% to 2.6% over full-data training, but the paper doesn't mention how many seeds were run or how much variance there is across runs. The gains also look less consistent when more data is pruned, for instance, at 40% retention, Moirai actually performs worse than random selection on both metrics. Having repeated runs alongside a per-task or per-dataset breakdown of the main GIFT-Eval results would make it much easier to tell whether these improvements are statistically solid and hold up across datasets.
-
It also seems like an open question whether this method beats simpler, practical alternatives for data selection. Right now, the main comparisons are limited to random selection, full-data training, or ablations of the authors' own filtering rule. Since the core idea closely follows existing loss-based pruning methods, it would be nice to see clearer evidence that a pretrained TSFM reference model is actually necessary. For example, how well would the exact same stratified selection pipeline work if windows were simply scored by a normalized seasonal-naive loss, or by an existing time-series quality rater like TSRating?
Questions for authors
-
Figure 4 reports rank correlations over the global candidate pool, whereas the actual selection algorithm relies on within-source percentiles. Do the reference models also agree on within-source rankings and on which windows actually get kept or dropped? I think this might help clarify whether the reported correlations explain the transfer results.
-
Could you specify how the "native masked loss" is formulated per window for each of the reference models? Additionally, since your theoretical analysis assumes MSE, which parts of that interpretation still hold for the quantile and negative log-likelihood losses used in practice?
5 comments
R3W2
We add the normalized seasonal-naive loss alternative using the exact same stratified selection pipeline at 80% retention. Table below shows that RFL improves both MASE and CRPS across all architectures, which is a clearer evidence that a pretrained TSFM reference model is useful for data selection.
| Method | Chronos/ MASE | Chronos/ CRPS | Moirai/ MASE | Moirai/ CRPS | TimesFM/ MASE | TimesFM/ CRPS | Timer/ MASE | Timer/ CRPS |
|---|---|---|---|---|---|---|---|---|
| Seasonal | 0.798 | 0.560 | 1.031 | 0.703 | 0.832 | 0.569 | 0.938 | 0.778 |
| Ours | 0.777 | 0.542 | 1.005 | 0.680 | 0.804 | 0.554 | 0.909 | 0.742 |
| Full | 0.796 | 0.555 | 1.017 | 0.698 | 0.813 | 0.558 | 0.978 | 0.788 |
Moreover, TSRating targets task-specific training or fine-tuning through LLM pairwise judgments and a meta-learned rater, and selects a subset. In contrast, we focus on pretraining time-series foundation models. Scaling TSRating’s LLM annotation and meta-training to our more than 100 source datasets would introduce substantial computational resource cost and time cost, whereas RFL requires only one frozen forward pass of a lightweight model.
R3Q1
The reference models also agree on within-source rankings and on which windows actually get kept or dropped. We additionally report the Pearson correlation between source-wise percentile vectors as shown in Figure 4. TimesFM and Chronos-Bolt agree strongly (0.879), while Moirai agrees weakly with TimesFM (0.182). We also compare retained-window overlap at 80% retention using source-wise sorting. Among the windows retained by TimesFM, 92.04% are also retained by Chronos-Bolt, whereas Moirai’s overlap with TimesFM is consistent with that of random selection (80%). Thus, the transfer results are consistent with both within-source rankings and selected-window agreement.
R3Q2
(1) Per-window formulation of the native masked loss. Appendix B now specifies all four objectives. Chronos-Bolt uses the normalized pinball loss. Timer uses squared errors and TimesFM combines point MSE with the sum of the pinball loss. Moirai uses the negative log-likelihood of each observed future value.
(2) Interpretation for quantile and negative log-likelihood losses. Appendix B extends the gradient analysis to both objectives.
Quantile loss. The gradient depends on the sign of the residual and on the quantile level, and the corresponding norm bound contains no residual-magnitude term (Appendix B.1).
Negative log-likelihood. The gradient is a responsibility-weighted combination of the component likelihood gradients (Appendix B.2). With a shared fixed variance, Gaussian NLL is an affine transformation of MSE, so the MSE interpretation holds exactly. With a predicted scale or mixture components, the update additionally reflects predictive uncertainty and distributional fit, which goes beyond the MSE interpretation.
Dear Reviewer L29t, as the discussion is coming to an end, we would like to check whether your concerns have been adequately addressed. We look forward to hearing from you. Best regards
Thanks for the careful review and the insightful questions.
R3W1
(1) Multiple runs and variance. The main-text results use one training seed because each pretraining run incurs substantial GPU cost. We now report three runs with different seeds, comparing Full with Random and Ours at 80% retention. The table below directly reproduces the mean and standard deviation results. Ours has lower mean MASE and CRPS than both baselines for all target architectures, supporting the reliability of the improvements across runs.
(2) Consistent gains. Moirai’s result at 40% retention is because a narrower central interval can reduce data diversity, while random selection gains an advantage through broader pattern coverage. Ayed and Hayou [M1] theoretically identify generalization degradation induced by score-based pruning under strong compression in their analyzed settings. Moreover, we emphasize that our main contribution is outperforming full-data pretraining by retaining 80% of the candidate pretraining windows.
[M1] Fadhel Ayed and Soufiane Hayou. Data Pruning and Neural Scaling Laws: Fundamental Limitations of Score-Based Algorithms. Transactions on Machine Learning Research, 2023.
(3) Per-task breakdown. We additionally report the results on the long-horizon and short-horizon forecasting tasks in GIFT-Eval. Table below reports this breakdown and Ours improves both metrics over Full and Random on long-horizon and short-horizon tasks.