acceptodds
Under review as a conference paper at ICLR 2027

When, Not How Much: Evaluating Time-Series Foundation Models on Sparse Events

Abstract

Pretrained time-series foundation models (TSFMs) are evaluated as forecasters of future values, yet for sparse series many decisions depend only on which future periods contain activity. Standard benchmarks do not assess this. On five sparse datasets, we rank positions within forecast windows that contain both events and zeros. The released point forecasts of 12 TSFMs improve chance-corrected average precision over training-free references by at most 0.031, and in chance-corrected AUC the median TSFM falls below them on every dataset. With event supervision, linear probes of six frozen backbones improve on their backbone's point forecast in 29 of 30 backbone–dataset pairs. Averaging the predicted quantiles instead of taking their median improves the ranking of most TSFMs that forecast the median, and on two datasets the strongest such outputs rival the probes. The probes' advantage over raw-context learners depends on the dataset, and under the same probe, pretrained features outperform randomly initialized ones for five of six backbones. For sparse-event ranking, released point forecasts thus add little over simple references, whereas lightweight event heads on frozen TSFMs rank events better than these forecasts, and the best of them exceed gradient-boosted trees trained on the raw context on three of the five datasets. More broadly, assessing pretrained forecasters on tasks beyond value forecasting requires reporting their outputs, supervised probes of their representations, and raw-context and randomized controls side by side, since each supports a different conclusion.

Primary area:
learning on time series and dynamical systems
Keywords:
time series foundation models, sparse time series, representation learning, linear probing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_91D2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates TSFMs as sparse event forecasters, treating forecasting as a binary detection or window-level ranking problem rather than regression. The paper compares various forecasting models including, 12 pretrained TSFMs, linear probes on frozen and random initialized backbones, and raw-context models. They transform model forecasts into a position score used to rank event likelihood using various methods including directly using the mean/median point forecasts and extracting a score from the quantile forecasts. Window-level ranking is measured through normalized average precision and AUROC and is evaluated on sparse subsets of 5 common real-world datasets. They found that all forecasting methods evaluated performed poorly with only minor gains compared to a random ordering. In particular, directly using TSFM point forecasts often performed worse than a supervised linear probe on frozen backbones or a LightGBM model trained on the raw context data.

Critical strengths
  • The paper is very thorough in their evaluation with well-thought out TSFM baselines and in-depth sensitivity analysis in the appendix (e.g. probes on random init backbones and comparing various position score methods that utilize the probabilistic output of TSFMs). There was considerable effort in the design of the experiments and covers many of their bases and potential follow up questions.
  • The main paper is well written and concise. It is easy to follow their experimental setup.
Critical weaknesses
  • This paper is written as an occurrence forecasting evaluation paper from a TSFM perspective failing to incorporate pre-existing sparse-field research. For example, all 5 evaluated datasets are subsets of standard TSFM benchmarking datasets. While M5 and Favorita overlap with intermittent-demand literature, RAF (Teunter & Duncan 2009; Nikolopoulos et al. 2011; Türkmen et al. 2021), Car Parts (Snyder et al. 2012), and Auto (Syntetos & Boylan 2005) are left out. These datasets may not meet the paper’s 364 daily/336 hourly context length or horizon (H=64) targets, but this seems to be a TSFM-style design choice rather than a fundamental limitation. The authors fail to label the datasets using ADI/CV², the standard dataset descriptors that allow a better understanding of when different ranking methods fail. Additionally, the paper only compares to standard time series models (logistic regression and LightGBM), but doesn’t compare to occurrence-specific baselines like Seasonal iETS (Svetunkov & Boylan 2023) and dynamic count models (Snyder et al. 2012).

  • The finding that TSFMs do not provide substantial improvement over reference methods or even random ranking is a relevant finding, but it could be confounded with a lack of signal in the datasets. While better baselines or additional real-world datasets can help alleviate this problem, I wonder if a controlled setting with synthetically generated data could give a more accurate picture of TSFM performance.

At its current state the paper feels blind to the field it is investigating. However, with minor additions (see above) that align it closer to existing occurrence forecasting research, I may reconsider an improved score.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 91D2

Thank you for pointing us to the intermittent-demand literature. We agree that it is an important context for this work and added each element you name.

W1 — ADI/CV² descriptors

Concern: the datasets are not labeled with ADI and CV².

Answer: We added both descriptors. In every dataset, more than 85% of the evaluated contexts are intermittent or lumpy. Table 2 of the paper now gives the median ADI and CV² per dataset, and Appendix A.4 the full descriptors.

Table 1. Evaluated contexts, with the 1.32 and 0.49 cut-offs of Syntetos et al. (2005).

AmazonFavoritaM5NYC taxiWeb traffic
Median ADI12.132.483.604.203.08
Median CV²1.000.450.290.220.52
Intermittent (%)18.349.182.788.645.8
Lumpy (%)81.736.813.911.346.6

Appendix E.4 repeats the main comparisons within each ADI/CV² category. Within the 16 dataset–category pairs with enough windows, the median linear probe exceeds the point forecast in 15 pairs in AP skill and in all 16 in AUC skill. The median pretrained probe exceeds its randomized copy in all 16 under both metrics. Raw LightGBM leads all ten probes in every web-traffic category. Point forecasts gain little over the training-free reference in intermittent contexts. They trail it on web-traffic smooth and erratic contexts.

W1 — Occurrence-specific baselines (iETS and dynamic count models)

Concern: no comparison with occurrence-specific models such as iETS (Svetunkov & Boylan, 2023) and the dynamic count models of Snyder et al. (2012).

Answer: We added the iETS occurrence model and the dynamic negative-binomial model, scored on all windows of the main study (Appendix F.1). The linear probes of all ten backbones exceed the two models in 87 of 90 AP comparisons. Each model is fit to the context of one window and gives an event probability for every target position.

  • Against the reference. iETS falls below the training-free reference on four of the five datasets, by 0.008 to 0.088 AP skill, and spans zero on Amazon. The negative-binomial model falls below it on all four count datasets (Amazon purchase amounts are not counts), by 0.005 to 0.184.
  • Against point forecasts. The 12 TSFM point forecasts exceed iETS in 41 of 60 AP comparisons and the negative-binomial model in 44 of 48.
  • Against probes. The three exceptions to 87 of 90 are against iETS on Amazon.

W1 — Intermittent-demand datasets

Concern: RAF, Car Parts, and Auto are left out.

Answer: We added all three (Appendix F.2). All three fall in the intermittent category by median ADI and CV². As in Snyder et al. (2012) and Türkmen et al. (2021), the last six months of each series form the target.

  • Probes. On RAF, the probes exceed their point forecasts for 7 of the 10 backbones, by a median of 0.076 AP skill.
  • Point forecasts. 35 of the 36 AP-skill intervals against the reference span zero; Sundial on Auto lies above, by 0.153.
  • Other comparisons. On Car Parts and Auto, all 20 probe-versus-forecast intervals span zero. Over the three datasets, 29 of 30 AP intervals against raw logistic span zero, as do all 30 against raw LightGBM and 29 of 30 pretrained-versus-randomized intervals. The negative-binomial model spans zero against the reference on all three datasets. iETS returns no feasible fit on most of these windows, so it is not scored on the monthly datasets.

Each series gives one window, so the trained methods see 279 to 4,500 training windows, against up to 100,000 in the main study. Hence, the main study uses daily and hourly sources, where long histories give many windows.

AuthorsOpenReview ↗

Response to Reviewer 91D2

W2 — Signal in the datasets

Concern: weak TSFM results could reflect a lack of signal in the data; a synthetic setting could help.

Answer: We agree that a controlled setting helps separate these explanations and added one (Appendix H). We adapt the synthetic arrival patterns of Türkmen et al. (2021) and add a probabilistic dropout variant. Under these patterns, our within-window ranking protocol rewards timing signal when it is present, and several TSFMs capture it. We score the point forecasts of all 12 TSFMs, raw logistic, and raw LightGBM.

  • No pattern (each position an event with probability 0.1): all methods score within 0.002 AP skill of zero.
  • Periodic and alternating gaps: both raw learners and seven of the 12 TSFMs reach at least 0.99 AP skill; Chronos-2, TimesFM-2.5, TimesFM-3, and Toto-2.0 rank every window perfectly.
  • Dropout (alternating gaps, each event kept with probability 0.5): Chronos-2 (0.509), Toto-2.0 (0.501), and TimesFM-3 (0.493) come close to raw logistic (0.518) and raw LightGBM (0.522); all 12 TSFMs trail both learners.

Timing signal is also present in the real datasets: the training-free reference, a seasonal profile of the context, reaches 0.103 AP skill on Favorita and 0.185 on NYC taxi, and the best probe exceeds it by 0.092 on Favorita and 0.057 on web traffic (Table 3 of the paper). Amazon is the exception, where no point forecast or probe gains more than 0.006 over the reference.

W3 — Connection to the intermittent-demand literature

Answer: Related work (Section 2) now covers Croston's method, TSB, iETS, and the dynamic count models of Snyder et al. (2012), and the methods state which are scored. Together with ADI/CV² descriptors, the two occurrence models, and the three monthly benchmarks above, the evaluation is now placed within this literature.

Reviewer_91D2OpenReview ↗

Reviewer Response to Rebuttal

I am not fully satisfied yet by the author's rebuttal. The inclusion of the ADI/CV² descriptors are appreciated however the main-text lacks proper analysis of skill w.r.t. these descriptors. When/where these methods fail is never addressed (e.g. do pretrained models perform better on smooth/erratic contexts?). Also, the confounding issue where the real-world data potentially lacks signal is not fully resolved by the synthetic experiments. It's hard to identify raw performance without a ceiling to rank against. The occurrence baseline implementation is weak (e.g. iETS only had a single configuration rather than validation fitted), but I will not mark against that.

AuthorsOpenReview ↗

Response to Reviewer 91D2

Thank you for the follow-up and the concrete suggestions, which have made the paper more informative.

1. Skill by ADI/CV² category. The new Section 4.4 analyzes skill by demand category in the main text. In short, the category does not change our conclusions, and no category consistently favors the TSFM point forecasts:

  • On M5, 7 of the 12 forecasts beat the training-free reference in smooth contexts, against 2 in intermittent contexts. On web traffic, all 12 trail it in smooth and erratic contexts.
  • In intermittent contexts, no forecast gains more than 0.026 AP skill on any dataset.
  • On three datasets, nearly all methods score higher in smooth and erratic contexts, the reference included. These contexts' targets contain more events.
  • The probe comparisons hold within categories. The median probe exceeds the point forecast in 15 of 16 dataset–category pairs in AP skill and in all 16 in AUC skill. The median pretrained probe exceeds its randomized copy in all 16.

2. Ceiling and signal in the real data. The synthetic experiments now include a ceiling (Section 4.1, Appendix H). Under dropout, an oracle that knows the generator reaches 0.529 AP skill and 0.890 AUC skill. Appendix H proves that this is the highest expected skill attainable from the context. Both raw learners come within 0.011 AP skill of this ceiling and the best TSFM, Chronos-2, within 0.020. The median TSFM lies 0.379 below it.

No ceiling can be computed on the real data, but the raw learners bound the attainable skill from below. Both exceed the median TSFM point forecast on every dataset under both metrics, with all 20 paired intervals above zero (Table 15). The real data thus contain timing signal that the point forecasts leave unused.

3. Occurrence baselines. Agreed. Section 4.1 now restricts the iETS and negative-binomial statement to the configurations we test. The Limitations state that these configurations were fixed in advance rather than selected on validation data.

We hope these additions address your remaining concerns and would be grateful if you would take them into account in your final assessment.

Reviewer_4CS7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the ability of time-series foundation models to rank sparse future events within a forecast window. Its central contribution is to distinguish three questions that are often conflated: whether a model’s released forecasts rank events effectively, whether its frozen internal representations are useful under event-specific supervision, and whether any representation value is attributable to pretraining rather than architecture or preprocessing alone.

The study evaluates twelve forecasters and probes six frozen backbones across five datasets. It includes raw-context logistic regression and LightGBM baselines, randomized-backbone controls, alternative reductions of probabilistic forecast outputs, missingness analyses, and several sensitivity checks. The principal empirical finding is appropriately nuanced. Released point forecasts generally deliver limited improvements over simple reference rankings. Event-supervised probes improve on their own backbone’s default forecast in nearly every comparison, but they do not consistently outperform direct supervised learning from the raw history. Randomized-backbone and source-overlap analyses provide evidence that pretraining contributes useful information in many settings, although the magnitude and practical significance of this benefit vary substantially across datasets and models.

Critical strengths

1. The evaluation separates three conceptually different claims.

The strongest aspect of the paper is its separation of forecast quality, supervised representation utility, and the benefit of pretraining. A supervised probe’s improvement over an unsupervised model output does not by itself establish that the pretrained representation is useful, because the probe has access to event labels that the original forecast did not use. The paper recognizes this and introduces several relevant controls, including raw logistic regression, raw-context LightGBM, randomized versions of the backbones, and supervised readouts of forecast outputs.

The results are also reported with appropriate qualifications. Probes improve on their own backbone’s point forecast in 29 of 30 model–dataset pairs, yet all six probed models trail LightGBM on Favorita and web traffic. Timer’s randomized features outperform its pretrained features on several datasets. The audit of disclosed pretraining sources is particularly useful: among 12 model–dataset pairs documented as excluded from pretraining, 11 have pretrained-versus-randomized confidence intervals above zero. Together, these controls make the study substantially more informative than a simple comparison claiming that foundation-model embeddings are useful.

2. The ranking protocol is carefully designed.

The within-window ranking formulation is well matched to the paper’s motivating question. It prevents methods from receiving credit merely for separating active series or windows from inactive ones, rather than identifying the timing of events within a given horizon.

The paper also treats the evaluation metric with unusual care. It derives a tie-aware permutation reference for average precision, reports AUC and equal-series-weighted results, and provides an interpretable top-eight event-count analysis. Importantly, the paper does not overinterpret positive AP skill as evidence of unequal marginal event probabilities when labels may be dependent.

The sensitivity analyses are similarly strong. Quantile averaging, training-size checks, missingness subsets, paired series-bootstrap intervals, and different weighting schemes reveal exceptions that a less careful study might have hidden. The finding that a simple alternative reduction of forecast outputs can rival the probes on some datasets is especially important and improves the credibility of the paper’s overall analysis.

3. The paper reports negative and mixed findings transparently.

The paper does not force the results into a universal foundation-model advantage. It reports cases where randomized features are competitive, where direct LightGBM learning is substantially stronger, and where processed outputs rival or exceed internal-feature probes. This transparency materially increases the value of the work as an evaluation paper and provides a more realistic picture of when pretrained representations may or may not be useful.

Critical weaknesses

1. The added value of pretrained internal features is not fully isolated.

The central comparison against raw logistic regression is not fully matched. The raw learner and the probed-model pathways use different normalization procedures, including backbone-specific per-window transformations. The NYC taxi diagnostic illustrates why this matters: training raw logistic regression on the test-year population increases AP skill by 0.086, nearly matching the reported median probe advantage of 0.087.

This diagnostic changes both normalization and the training population, so it does not establish that preprocessing explains the probe advantage. Nevertheless, it shows that the existing comparison cannot cleanly attribute that advantage to pretrained features. Stronger direct baselines using matched per-window normalization, binary event histories, time since last event, recent event frequency, and simple seasonal indicators would better establish the representations’ practical incremental value.

The comparison between internal features and forecast outputs is also not fully controlled. The supervised output readouts appear to use different training populations, sample sizes, validation procedures, or model-selection rules from the main probes. Several forecasters, including Moirai-2.0, are not probed. Moreover, averaging forecast quantiles outperforms most probes on M5 and NYC taxi. The current results therefore establish that supervised probes improve selected models’ default output reductions more strongly than they establish that internal features are inherently more useful than properly processed probabilistic outputs under matched supervision.

2. Missingness-as-zero creates a significant construct-validity problem.

A substantial part of the web-traffic evaluation measures recorded availability as well as event occurrence. Forty-five percent of scored target positions are missing observations stored as zero, 68% of zero labels arise from missingness, and 91% of windows contain at least one missing target day. Under this construction, a learner may receive credit for predicting the data-recording process rather than actual inactivity.

Equal treatment of methods does not resolve this issue because the concern is not comparative fairness alone. It concerns what the target variable represents. This is particularly important because web traffic produces both a large probe-over-forecast gain and a substantial LightGBM-over-probe result.

The complete-window sensitivity analysis is useful but not decisive. It retains only 1,907 of 41,816 windows, while all models remain trained on labels in which missing observations are treated as zeros. In that subset, the median LightGBM–probe gap falls from 0.115 to 0.039, and only three of six pretrained-versus-randomized intervals remain above zero. These results do not eliminate the paper’s findings, but neither do they establish that the full-data effect sizes reflect prediction of genuine inactivity.

Related issues occur elsewhere in weaker forms, including unobserved stockouts in Favorita, truncation of fractional sales, and zero-filling before the first recorded activity. These choices may be defensible for a conditional ranking benchmark, but the conclusions should remain scoped to recorded nonzero observations unless the authors can validate the event labels more directly.

3. The presentation of the probe design is too compressed.

The distinction among forecast outputs, supervised probes, and the contribution of pretraining is the paper’s main conceptual contribution, but Table 1 does not make these distinctions sufficiently accessible.

In particular, “pretrained vs. randomized linear probe” may be misread as randomizing the probe. In fact, both logistic probes are trained on event labels; the frozen backbone supplying their features differs in whether it uses pretrained or randomly initialized weights. Similarly, a probe’s improvement over the original forecast reflects both access to internal features and event-specific supervision. It does not, by itself, identify the value of pretraining.

The paper should explain immediately before Table 1 that a linear probe is a logistic classifier trained to map frozen backbone features to event scores for the 64 future positions. Table 1 would be much clearer with columns such as: a) Input to the learner; b)Frozen component; c) Trainable component; d)Use of event labels; e)Claim supported by the comparison; f)Claim not supported by the comparison.

A short plain-language interpretation of each comparison would make the paper’s primary experimental logic substantially easier to follow.

Questions for authors
  1. Matched direct baselines: Can the authors repeat the raw-logistic comparison using the same per-window normalization applied in each probed backbone? Could they also include direct learners using binary event history, time since the last event, recent event counts, and calendar or seasonal features, while preserving the original training years and series splits?

  2. Separating preprocessing from population shift: In the NYC taxi diagnostic, how much of the 0.086 AP-skill improvement comes from normalization and how much comes from training on the test-year population? Can these two changes be evaluated separately?

  3. Matched output readouts: Can the supervised point-forecast and quantile-output readouts use exactly the same training windows, sample cap, validation set, regularization search, and selection objective as the internal-feature probes? This would provide a cleaner test of whether hidden features contain useful event information beyond the model’s output distribution under equal supervision.

  4. Coverage of stronger output models: Can the authors probe at least one strong model that is currently omitted, particularly Moirai-2.0? If this is not technically possible, can they explain what access or architectural constraint prevents a matched probe?

  5. Observed-label web-traffic evaluation: Can missing web-traffic targets be represented as unknown and excluded from both training losses and evaluation metrics, rather than assigned zero labels? Can the main probe, randomized-backbone, raw-logistic, and LightGBM comparisons be refitted under this protocol?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4CS7

Thank you for the constructive and detailed review; we ran each analysis you proposed.

Q1 / W1 (first paragraph) — Matched direct baselines

Concern: raw logistic regression did not use the probes' per-window normalization, and stronger direct learners on event features were missing.

Answer: Raw logistic now uses per-window standardization followed by training-set feature standardization, and is the baseline in all main comparisons (Section 4.2). The ten probes exceed it in 17 of 50 AP comparisons and trail it in 22. The incremental value over direct learning depends on dataset and backbone. Pretrained probes exceed their randomized copies in 45 of 50. All refits keep the original years, splits, training windows, and selection rule.

  • Backbone-specific normalization. Logistic regression with each backbone's own input transformation changes the comparison little: the probes exceed it in 19 of 50 AP comparisons and trail it in 24.
  • Event features. Logistic regression on binary event history, time since the last event, and recent event counts (occurrence logistic) is within 0.007 AP skill of the median probe on every dataset. A variant that also receives weekday and hour of each target position, which the TSFMs do not, exceeds all ten probes on Amazon and M5.

Table 1. Median probe − baseline AP skill over the ten backbones, equal-window weighting; intervals above / spanning / below zero (Appendix E.10).

Probe −AmazonFavoritaM5NYC taxiWeb traffic
raw logistic−0.002 (0/5/5)−0.002 (5/0/5)0.001 (5/1/4)−0.005 (1/2/7)0.006 (6/3/1)
logistic, backbone normalization−0.002 (0/3/7)−0.002 (5/0/5)0.001 (5/1/4)−0.005 (3/1/6)0.005 (6/2/2)
occurrence logistic−0.006 (0/0/10)0.007 (6/1/3)−0.002 (2/2/6)−0.006 (1/1/8)0.002 (5/1/4)
occurrence + calendar−0.008 (0/0/10)−0.012 (1/1/8)−0.017 (0/0/10)−0.007 (1/1/8)0.001 (5/1/4)

Q2 — NYC taxi: normalization versus training population

Concern: how much of the 0.086 AP-skill improvement comes from normalization and how much from training on the test-year population?

Answer: The submitted 0.086 compared a logistic fit trained on the 2024 test year (0.2229) with the 2022 raw logistic fit (0.1367), which was trained with erroneous preprocessing (see the general comment). Corrected, the 2022-trained fit reaches 0.2301, above the test-year fit. The 0.086 therefore reflects the preprocessing error, not a benefit of normalization or of test-year training.

A 2×2 design trains and selects on 2022 or 2024, with or without per-window standardization, holding out the same location folds with matched training counts per location (Appendix E.6).

Table 2. AP-skill effects, equal-window weighting; paired 95% series-bootstrap intervals.

ChangeAP skill
Per-window standardization, trained on 2022+0.0032 [0.0010, 0.0055]
Per-window standardization, trained on 2024+0.0087 [0.0055, 0.0118]
2024 instead of 2022, without per-window standardization−0.0008 [−0.0035, 0.0021]
2024 instead of 2022, with per-window standardization+0.0047 [0.0020, 0.0072]

Per-window standardization helps in both years; training on the test year helps only together with it (interaction +0.0055 [0.0029, 0.0081]). AUC skill shows the same pattern. The 2024-trained cells are retrospective diagnostics, since labels at other locations share citywide shocks with the evaluated ones.

AuthorsOpenReview ↗

Response to Reviewer 4CS7

Q3 / W1 (second paragraph) — Matched output readouts

Concern: the output readouts used different training and selection than the probes, so internal features were not compared with outputs under equal supervision.

Answer: Under matched supervision, the internal features rank events better than the outputs. The readout now uses the probe's training-window sampling, validation windows, L2 grid, and selection rule, on the point forecast and, where available, nine quantiles (Appendix G.3). The selected readout also chooses its input transform on validation data, which the probe cannot.

Table 3. Median AP skill over backbones, equal-window weighting; in parentheses, paired 95% series-bootstrap intervals above / spanning / below zero.

ContrastBackbonesAmazonFavoritaM5NYC taxiWeb traffic
Probe − readout, selectedoriginal six0.004 (5/1/0)0.016 (6/0/0)0.009 (6/0/0)0.003 (4/1/1)0.016 (6/0/0)
added four0.002 (2/2/0)0.007 (4/0/0)0.002 (3/0/1)0.017 (4/0/0)0.019 (4/0/0)
Readout − point forecastoriginal six0.002 (3/2/1)0.062 (6/0/0)0.021 (6/0/0)0.015 (6/0/0)0.031 (6/0/0)
added four0.002 (2/2/0)0.061 (4/0/0)0.023 (4/0/0)0.024 (4/0/0)0.029 (4/0/0)

The probe exceeds both readouts for all ten backbones on Favorita and web traffic, and the selected readout for nine on M5, eight on NYC taxi, and seven on Amazon, under AP and AUC skill. The readout is ahead only for TimesFM-2.5 on M5 and Chronos-2 on NYC taxi.

Adding the quantiles raises the readout by a median of at most 0.011 AP skill. The best probe exceeds the strongest readout on Amazon, Favorita, and web traffic and is level with it on M5; on NYC taxi, the Chronos-2 readout exceeds every probe.

Q4 — Probing Moirai-2.0 and other strong forecasters

Concern: strong output models such as Moirai-2.0 were not probed.

Answer: We now probe Moirai-1.1, Moirai-2.0, TimesFM-2.5, and TimesFM-3 with the same procedure as the original six (Section 4.2).

The Moirai-2.0 probe exceeds its own point forecast by 0.006–0.069 AP skill and its randomized counterpart by 0.007–0.060 across the five datasets, with all paired intervals above zero under AP and AUC skill. It gives the highest probe skill of all ten on M5 (0.077), exceeding raw LightGBM by 0.013 [0.012, 0.014].

AuthorsOpenReview ↗

Response to Reviewer 4CS7

Q5 / W2 — Web-traffic missingness

Concern: missing days stored as zeros may make the task partly a prediction of recording, and the complete-window check kept all models trained on zero-filled labels.

Answer: Following your suggestion, we treat missing target days as unknown and exclude them from training, validation selection, and scoring. Under this protocol, every main probe interval keeps its class. The LightGBM gap shrinks from 0.112 to 0.033 AP skill (Appendix E.5). We refit the pretrained and randomized probes of all ten backbones, raw logistic, and raw LightGBM, and score each window on its observed days (30,765 of 41,816 windows).

Table 4. Median AP skill over the ten backbones, equal-window weighting; intervals above / spanning / below zero. "Original models" were trained with missing days as zeros and are scored on observed days.

ContrastZeros, all windowsZeros, retained windowsUnknown, original modelsUnknown, refit
LightGBM − linear probe0.112 (10/0/0)0.075 (10/0/0)0.038 (10/0/0)0.033 (10/0/0)
Linear probe − point forecast0.048 (10/0/0)0.024 (10/0/0)0.023 (10/0/0)0.025 (10/0/0)
Pretrained − randomized0.016 (8/1/1)0.009 (8/1/1)0.011 (8/1/1)0.013 (8/1/1)
Linear probe − raw logistic0.006 (6/3/1)0.000 (3/4/3)0.002 (3/7/0)−0.003 (0/7/3)
  • LightGBM gap. It shrinks mostly through scoring rather than refitting (0.038 for the original models). Every probe still trails raw LightGBM.
  • Main comparisons. Under equal-series weighting, the pretrained-versus-randomized intervals of Timer (AP skill), Toto-2.0, and Moirai-1.1 move from above zero to spanning zero.
  • Raw logistic. The probes and raw logistic are on par, with 7 of 10 intervals spanning zero.

Because the source cannot distinguish days with zero views from data not yet loaded, we keep the zero-filled evaluation as the main one and scope the web-traffic conclusions to recorded activity. Appendix A.2 now also documents the other sources of zeros you name: unrecorded stock-outs in Favorita, fractional sales below one unit truncated to zero (0.4% of source records), and zeros before a series' first recorded activity.

W3 — Presentation of the probe design

Concern: Table 1 compresses the distinctions between outputs, probes, and pretraining.

Answer: Section 3 now defines the linear probe where Table 1 of the paper is introduced: a logistic classifier, trained on event labels, that maps the frozen representation to an event score for each target position. Table 1 now has an Event labels column, and its caption states that in every probe comparison the backbone is frozen and only the probe is trained. The matched readouts in Q3 separate the two sources of the probe's gain: supervision on the outputs (readout − forecast) and the internal features (probe − readout).

Reviewer_Bmk4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the evaluation of Pretrained Time-Series Foundation Models (TSFMs) on intermittent and sparse series, the core idea of the methodology is where decisions depend primarily on whether an event occurs rather than its quantitative magnitude.

The authors formalize this problem as within-window event ranking: basically given a history context xw∈RCx_w \in \mathbb{R}^C (they consider C=364C=364 daily or 336336 hourly observations), rank the target horizon positions h∈{1,…,H}h \in \{1, \dots, H\} (H=64H=64) by their probability of containing a positive event (yw,h=1(yw,h>0)y_{w,h} = 1(y_{w,h} > 0)). Evaluation is conducted on target windows containing mixed labels (0<kw<H0 < k_w < H, where kw=∑h=1Hyw,hk_w = \sum_{h=1}^H y_{w,h}).

Critical strengths

Current benchmarks in TSFM usually do not take into account event ranking, this paper shows how to formalise the within-window event ranking where zero-heavy windows are largely absent from standard evaluations. The derivation of the exact expectation for threshold (Average Precision) AP under label permutation with tie blocks is exact and handles tied predictions without order-dependent randomization.

The most innovative contribution I found is how the authors attempted to implement a principled set of controls to isolate representation utility vs. probe capacity and pretraining knowledge. By contrasting pretrained linear probes against (i) point forecasts, (ii) raw-context logistic regression, (iii) raw-context LightGBM, and (iv) randomized backbone probes, the authors establish clear boundaries on what pretrained representations actually provide.

Critical weaknesses

The authors note that probing was restricted to single-pass models fixed prior to experiment execution, probing these top-performing architectures would provide a more complete assessment of whether modern state-of-the-art forecasters. In particular they omit Moirai-1.1, Moirai-2.0, TimesFM-2.5, and TimesFM-3. Looking at table 3, Table 15, we see Moirai-2.0 and TimesFM-2.5 yield the strongest point forecasts and quantile averages on several datasets.

Questions for authors

Would the authors be able to explain Why were Moirai-2.0 and TimesFM-2.5/3.0 excluded from probing despite being single-pass models (Table 11)? Would probing their representations narrow or widen the gap to direct gradient-boosted trees (LightGBM) on datasets like Favorita and Web Traffic?

Also have you tested lightweight parameter-efficient fine-tuning (e.g., LoRA or unfreezing the last layer) for event classification? I am curious to see how much additional performance is gained when adapting backbone weights directly to the event detection objective compared to linear probing on frozen features?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Bmk4

Thank you for the review and the insightful questions on stronger backbones and adaptation.

Q1 / W1 — Probing Moirai-1.1, Moirai-2.0, TimesFM-2.5, and TimesFM-3

Concern: strong forecasters were not probed; would probing them narrow the gap to raw LightGBM on Favorita and web traffic?

Answer: We now probe all four with the same procedure as the original six, and the main study reports all ten probed backbones together (Section 4.2). The original six were the backbones our initial inference setup had supported for representation extraction; the final setup supports all ten. The added probes narrow the LightGBM gap slightly but do not close it on Favorita or web traffic.

Over all ten backbones, the median AP-skill deficit to raw LightGBM is 0.014 on Favorita and 0.112 on web traffic, against 0.020 and 0.115 for the original six. On M5, Moirai-2.0 gives the highest probe skill of all ten, ahead of raw LightGBM by 0.013 [0.012, 0.014].

Table 1. AP skill, equal-window weighting. Best probes are descriptive test-set maxima.

DatasetRaw LightGBMBest of original six probesBest of added four probes
Favorita0.1960.194 (Sundial)0.192 (Moirai-2.0)
M50.0640.076 (Chronos-2)0.077 (Moirai-2.0)
Web traffic0.1770.073 (Sundial)0.070 (TimesFM-3)

The added probes exceed their own point forecasts in all 20 backbone–dataset pairs and their randomized counterparts in 19 of 20.

Q2 — Adapting backbone weights to the event objective

Concern: how much does adapting backbone weights add over a frozen linear probe?

Answer: Following your suggestion to unfreeze the last layer, we train the final encoder block of Chronos-2 and Moirai-2.0 (9.5 and 1.8 million parameters) jointly with the logistic probe on the event labels of Favorita and NYC taxi (Appendix E.9). The gain is dataset-dependent: adaptation adds 0.012 AP skill on Favorita and does not improve on the frozen probe on NYC taxi.

Table 2. AP skill, equal-window weighting; 95% paired bootstrap intervals over series.

BackboneDatasetFrozenAdaptedAdapted − frozenAdapted − raw LightGBM
Chronos-2Favorita0.1930.205+0.012 [0.012, 0.013]+0.010 [0.009, 0.011]
Moirai-2.0Favorita0.1920.204+0.012 [0.012, 0.013]+0.008 [0.007, 0.009]
Chronos-2NYC taxi0.2240.221−0.003 [−0.004, −0.002]−0.005 [−0.008, −0.003]
Moirai-2.0NYC taxi0.2090.208−0.002 [−0.002, −0.001]−0.018 [−0.022, −0.015]

On Favorita, adaptation gains on each of the five splits and moves both probes from below raw LightGBM to above it. On NYC taxi, both adapted probes trail raw LightGBM. AUC skill shows the same pattern.

AuthorsOpenReview ↗

Response to Reviewer Bmk4

Thank you again for your positive assessment and for your questions on stronger backbones and on adapting the backbone weights. Both are now part of the paper: the main study probes all ten backbones, including the four you named (Section 4.2), and Appendix E.9 adapts the backbone weights to event labels. Following the discussion with the other reviewers, we have uploaded a second revision, and the general comment summarizes its changes.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Changes in the Revised PDF

We thank all reviewers for their detailed and constructive feedback, which we have implemented in the revised PDF.

Changes in the revision

  • Ten probed backbones. As requested by Bmk4 (Q1, W1) and 4CS7 (Q4), we now also probe Moirai-1.1, Moirai-2.0, TimesFM-2.5, and TimesFM-3; all ten form the main study (Section 4.2).
  • Matched output readouts. Addressing 4CS7 (Q3, W1), the output readouts are now trained with the same training windows, validation windows, and selection rule as the probes (Appendix G.3).
  • Direct baselines. Addressing 4CS7 (Q1, W1), raw logistic now applies per-window standardization, and we add logistic regression with each backbone's own normalization and on event-history and calendar features (Appendix E.10).
  • NYC taxi training year. Addressing 4CS7 (Q2), a 2×2 design separates per-window standardization from test-year training (Appendix E.6).
  • Web-traffic missingness. As proposed by 4CS7 (Q5, W2), we refit all probes and raw learners with missing target days excluded (Appendix E.5), and Appendix A.2 documents the sources of zeros.
  • Probe design. Following 4CS7 (W3), Section 3 defines the linear probe with Table 1, which now marks which score uses event labels.
  • Intermittent-demand literature. As suggested by 91D2 (W1, W3), Related work (Section 2) covers Croston's method, TSB, iETS, and dynamic count models. We add:
    • ADI and CV² per dataset (Table 2, Appendix A.4) and results by demand category (Appendix E.4);
    • iETS and the dynamic negative-binomial model of Snyder et al. (2012) (Appendix F.1);
    • the RAF, Car Parts, and Auto datasets (Appendix F.2).
  • Synthetic occurrence patterns. Addressing 91D2 (W2), Appendix H tests whether the protocol rewards timing signal.
  • Backbone adaptation. Addressing Bmk4 (Q2), we train the final block of Chronos-2 and Moirai-2.0 with the probe (Appendix E.9).

Correction on NYC taxi. Out-of-period timestamps in the source files extended the time grids of all years. The effect was largest on the 2022 training grid, where it changed which locations passed the sparsity filter and how training windows were aligned. The TSFM forecasts and probes were computed on the corrected grids; the raw-learner fits predated the correction and are now refit. With corrected fits, raw logistic without per-window standardization changes from 0.1367 to 0.2301 AP skill and raw LightGBM from 0.2243 to 0.2261. Raw logistic as now reported reaches 0.2281 (Table 18). The large taxi probe advantage over raw logistic in the submission therefore does not hold: the median probe trails raw logistic by 0.005 (Table 4). Forecast, probe, and pretrained-versus-randomized results are unaffected.

We hope the revision addresses the reviewers' concerns and are happy to answer further questions. We thank the reviewers again for their time.

AuthorsOpenReview ↗

Summary of the discussion and second revision

We thank the reviewers for the constructive discussion. We have uploaded a second revision that responds to the follow-up comments of reviewers 4CS7 and 91D2. Below, we summarize each reviewer's main requests and the changes in both revisions.

  • Reviewer Bmk4 asked about probing four further forecasters and adapting the backbone weights. The first revision adds probes of these four, so the main study covers ten backbones (Section 4.2). Appendix E.9 adapts backbone weights to event labels.
  • Reviewer 4CS7 asked for matched baselines, matched output readouts, more probed backbones, and a treatment of web-traffic missingness. The first revision added these, and the reviewer considered them addressed as analyses. Their follow-up concerned the framing, which the second revision changes:
    • The abstract, introduction, contributions, and discussion attribute most of the probe gain to event supervision and report that the probes are on par with raw logistic.
    • The abstract, Section 4.3, and the discussion limit the conclusion from the randomized control.
    • The occurrence + calendar baseline is part of the main scoreboard (Table 4).
    • The main text reports the observed-label web-traffic results in more detail, and we addressed the minor points.
  • Reviewer 91D2 asked us to connect the study to the intermittent-demand literature and to separate model performance from a possible lack of signal. The first revision added ADI/CV² descriptors with results by demand category (Appendix E.4), iETS and a dynamic count model, three intermittent-demand datasets, and synthetic occurrence patterns. The second revision adds:
    • a main-text analysis by demand category (Section 4.4);
    • an oracle ceiling for the synthetic experiments (Section 4.1, Appendix H) and a lower bound on the attainable skill in the real data (Table 15);
    • following the reviewer's note on the occurrence baselines, a restriction of these results to the tested configurations (Section 4.1, Limitations).

The paper contributes a task, five sparse evaluation datasets, and a scoring protocol for sparse-event ranking. It assesses the point forecasts of 12 TSFMs and probes ten frozen backbones. Output readouts, raw-context learners, and randomized backbones serve as controls that separate event supervision, frozen features, and pretrained weights.

Table numbering. The second revision adds Table 15, so the NYC taxi results cited in our first general response (raw logistic, 0.2281) are now in Table 19.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.