- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper evaluates TSFMs as sparse event forecasters, treating forecasting as a binary detection or window-level ranking problem rather than regression. The paper compares various forecasting models including, 12 pretrained TSFMs, linear probes on frozen and random initialized backbones, and raw-context models. They transform model forecasts into a position score used to rank event likelihood using various methods including directly using the mean/median point forecasts and extracting a score from the quantile forecasts. Window-level ranking is measured through normalized average precision and AUROC and is evaluated on sparse subsets of 5 common real-world datasets. They found that all forecasting methods evaluated performed poorly with only minor gains compared to a random ordering. In particular, directly using TSFM point forecasts often performed worse than a supervised linear probe on frozen backbones or a LightGBM model trained on the raw context data.
Critical strengths
- The paper is very thorough in their evaluation with well-thought out TSFM baselines and in-depth sensitivity analysis in the appendix (e.g. probes on random init backbones and comparing various position score methods that utilize the probabilistic output of TSFMs). There was considerable effort in the design of the experiments and covers many of their bases and potential follow up questions.
- The main paper is well written and concise. It is easy to follow their experimental setup.
Critical weaknesses
-
This paper is written as an occurrence forecasting evaluation paper from a TSFM perspective failing to incorporate pre-existing sparse-field research. For example, all 5 evaluated datasets are subsets of standard TSFM benchmarking datasets. While M5 and Favorita overlap with intermittent-demand literature, RAF (Teunter & Duncan 2009; Nikolopoulos et al. 2011; Türkmen et al. 2021), Car Parts (Snyder et al. 2012), and Auto (Syntetos & Boylan 2005) are left out. These datasets may not meet the paper’s 364 daily/336 hourly context length or horizon (H=64) targets, but this seems to be a TSFM-style design choice rather than a fundamental limitation. The authors fail to label the datasets using ADI/CV², the standard dataset descriptors that allow a better understanding of when different ranking methods fail. Additionally, the paper only compares to standard time series models (logistic regression and LightGBM), but doesn’t compare to occurrence-specific baselines like Seasonal iETS (Svetunkov & Boylan 2023) and dynamic count models (Snyder et al. 2012).
-
The finding that TSFMs do not provide substantial improvement over reference methods or even random ranking is a relevant finding, but it could be confounded with a lack of signal in the datasets. While better baselines or additional real-world datasets can help alleviate this problem, I wonder if a controlled setting with synthetically generated data could give a more accurate picture of TSFM performance.
At its current state the paper feels blind to the field it is investigating. However, with minor additions (see above) that align it closer to existing occurrence forecasting research, I may reconsider an improved score.
Questions for authors
See weaknesses.
4 comments
Response to Reviewer 91D2
W2 — Signal in the datasets
Concern: weak TSFM results could reflect a lack of signal in the data; a synthetic setting could help.
Answer: We agree that a controlled setting helps separate these explanations and added one (Appendix H). We adapt the synthetic arrival patterns of Türkmen et al. (2021) and add a probabilistic dropout variant. Under these patterns, our within-window ranking protocol rewards timing signal when it is present, and several TSFMs capture it. We score the point forecasts of all 12 TSFMs, raw logistic, and raw LightGBM.
- No pattern (each position an event with probability 0.1): all methods score within 0.002 AP skill of zero.
- Periodic and alternating gaps: both raw learners and seven of the 12 TSFMs reach at least 0.99 AP skill; Chronos-2, TimesFM-2.5, TimesFM-3, and Toto-2.0 rank every window perfectly.
- Dropout (alternating gaps, each event kept with probability 0.5): Chronos-2 (0.509), Toto-2.0 (0.501), and TimesFM-3 (0.493) come close to raw logistic (0.518) and raw LightGBM (0.522); all 12 TSFMs trail both learners.
Timing signal is also present in the real datasets: the training-free reference, a seasonal profile of the context, reaches 0.103 AP skill on Favorita and 0.185 on NYC taxi, and the best probe exceeds it by 0.092 on Favorita and 0.057 on web traffic (Table 3 of the paper). Amazon is the exception, where no point forecast or probe gains more than 0.006 over the reference.
W3 — Connection to the intermittent-demand literature
Answer: Related work (Section 2) now covers Croston's method, TSB, iETS, and the dynamic count models of Snyder et al. (2012), and the methods state which are scored. Together with ADI/CV² descriptors, the two occurrence models, and the three monthly benchmarks above, the evaluation is now placed within this literature.
Reviewer Response to Rebuttal
I am not fully satisfied yet by the author's rebuttal. The inclusion of the ADI/CV² descriptors are appreciated however the main-text lacks proper analysis of skill w.r.t. these descriptors. When/where these methods fail is never addressed (e.g. do pretrained models perform better on smooth/erratic contexts?). Also, the confounding issue where the real-world data potentially lacks signal is not fully resolved by the synthetic experiments. It's hard to identify raw performance without a ceiling to rank against. The occurrence baseline implementation is weak (e.g. iETS only had a single configuration rather than validation fitted), but I will not mark against that.
Response to Reviewer 91D2
Thank you for the follow-up and the concrete suggestions, which have made the paper more informative.
1. Skill by ADI/CV² category. The new Section 4.4 analyzes skill by demand category in the main text. In short, the category does not change our conclusions, and no category consistently favors the TSFM point forecasts:
- On M5, 7 of the 12 forecasts beat the training-free reference in smooth contexts, against 2 in intermittent contexts. On web traffic, all 12 trail it in smooth and erratic contexts.
- In intermittent contexts, no forecast gains more than 0.026 AP skill on any dataset.
- On three datasets, nearly all methods score higher in smooth and erratic contexts, the reference included. These contexts' targets contain more events.
- The probe comparisons hold within categories. The median probe exceeds the point forecast in 15 of 16 dataset–category pairs in AP skill and in all 16 in AUC skill. The median pretrained probe exceeds its randomized copy in all 16.
2. Ceiling and signal in the real data. The synthetic experiments now include a ceiling (Section 4.1, Appendix H). Under dropout, an oracle that knows the generator reaches 0.529 AP skill and 0.890 AUC skill. Appendix H proves that this is the highest expected skill attainable from the context. Both raw learners come within 0.011 AP skill of this ceiling and the best TSFM, Chronos-2, within 0.020. The median TSFM lies 0.379 below it.
No ceiling can be computed on the real data, but the raw learners bound the attainable skill from below. Both exceed the median TSFM point forecast on every dataset under both metrics, with all 20 paired intervals above zero (Table 15). The real data thus contain timing signal that the point forecasts leave unused.
3. Occurrence baselines. Agreed. Section 4.1 now restricts the iETS and negative-binomial statement to the configurations we test. The Limitations state that these configurations were fixed in advance rather than selected on validation data.
We hope these additions address your remaining concerns and would be grateful if you would take them into account in your final assessment.
Response to Reviewer 91D2
Thank you for pointing us to the intermittent-demand literature. We agree that it is an important context for this work and added each element you name.
W1 — ADI/CV² descriptors
Concern: the datasets are not labeled with ADI and CV².
Answer: We added both descriptors. In every dataset, more than 85% of the evaluated contexts are intermittent or lumpy. Table 2 of the paper now gives the median ADI and CV² per dataset, and Appendix A.4 the full descriptors.
Table 1. Evaluated contexts, with the 1.32 and 0.49 cut-offs of Syntetos et al. (2005).
Appendix E.4 repeats the main comparisons within each ADI/CV² category. Within the 16 dataset–category pairs with enough windows, the median linear probe exceeds the point forecast in 15 pairs in AP skill and in all 16 in AUC skill. The median pretrained probe exceeds its randomized copy in all 16 under both metrics. Raw LightGBM leads all ten probes in every web-traffic category. Point forecasts gain little over the training-free reference in intermittent contexts. They trail it on web-traffic smooth and erratic contexts.
W1 — Occurrence-specific baselines (iETS and dynamic count models)
Concern: no comparison with occurrence-specific models such as iETS (Svetunkov & Boylan, 2023) and the dynamic count models of Snyder et al. (2012).
Answer: We added the iETS occurrence model and the dynamic negative-binomial model, scored on all windows of the main study (Appendix F.1). The linear probes of all ten backbones exceed the two models in 87 of 90 AP comparisons. Each model is fit to the context of one window and gives an event probability for every target position.
W1 — Intermittent-demand datasets
Concern: RAF, Car Parts, and Auto are left out.
Answer: We added all three (Appendix F.2). All three fall in the intermittent category by median ADI and CV². As in Snyder et al. (2012) and Türkmen et al. (2021), the last six months of each series form the target.
Each series gives one window, so the trained methods see 279 to 4,500 training windows, against up to 100,000 in the main study. Hence, the main study uses daily and hourly sources, where long histories give many windows.