acceptodds
Under review as a conference paper at ICLR 2027

Beyond Static Leaderboards: Capability-Focused Evaluation with Structural Diversity for Time Series Foundation Models

Abstract

Static public benchmarks for time series foundation models (TSFMs) typically have fixed and narrow test distributions. Their repeated reuse encourages benchmark-specific tuning, making reported performance reflect adaptation to the fixed test set rather than genuine forecasting generalization. In addition, a single aggregate score cannot fully characterize model capabilities across distinct temporal structures and input relationships. We propose CaFE (Capability-Focused Evaluation), a refreshable evaluation extension grounded in original benchmark data that enriches static benchmarks with structural diversity and enriches single-score black-box evaluation with multi-dimensional, capability-oriented assessment. CaFE first profiles the original benchmark and defines eight Structural Features (SFs) that characterize its temporal structure. For each SF, it resamples the original data according to its structural support, removes edge-of-distribution samples, and augments the retained samples by varying the SF's salience, producing samples with different structural profiles. After filtering out out-of-distribution samples while preserving diversity, CaFE yields a refreshed evaluation set. TSFMs are then evaluated on this set using both conventional forecast-error metrics and intervention-response metrics, giving each model a per-feature capability profile alongside endpoint accuracy. Experiments on GIFT-Eval and FEV-Mini20 show that CaFE yields stable evaluation results and reveals per-feature strengths and weaknesses that a single aggregate score hides. Moreover, on genuine data filtered by structural condition, models selected by CaFE outperform those chosen by the aggregate leaderboard. CaFE therefore makes model selection conditional on the structural response of interest.

Primary area:
datasets and benchmarks
Keywords:
time series foundation models, live benchmark, time series forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_X1swOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The motivation of the work is that static time series benchmarks have narrow test distributions that can lead to overfitting / benchmark-specific tuning, and that single aggregation scores in these benchmarks cannot characterize model capabilities across specific temporal structures, and that they are not useful (enough) for model selection.

With that motivation, the authors propose CaFE - capability focused evaluation which enriches static benchmarks with structural diversity (through time series features), and further goes beyond a single aggregate score with a multi-dimensional assessment. This is done by defining structural features, profiling the benchmark and then augmenting it with samples along specific features.

They evaluate time series models on their proposed benchmark extension to give each model a per-feature capability profile, which allows quantifying the failure modes of each model better.

Critical strengths

I believe the two key points in the motivation of the paper are strong: That existing benchmarks (GIFT-EVAL, fev bench) are narrow and that it is really hard to make a model selection decision based on the aggregate scores on these benchmarks

The proposed high level idea is also strong - to profile benchmarks and give more dimensions of evaluation.

I do not think these constitute critical strengths that support acceptance, however. More details below.

Critical weaknesses

I summarize the critical weaknesses first, and then paste the raw review I wrote section-by-section as I read the paper.

SUMMARY

  • Sections 3 and 4 are extremely hard to follow. They lack crucial low-level implementation details, clear rationale, and definitions for key terms. The writing style and phrasing make it strongly look like heavy AI copy-editing. This makes text impossible to understand.
  • Core tables and figures are not useful enough (like Table 1 lacking citations, row/column definitions, or Table 2 being referenced in the main text but hidden in the appendix) or poorly explained. The results sections just dump low-level stats and correlations without summarizing what they actually mean. And since the benchmark section is not clearly written, I'm not able to get anything concrete from the results section.
  • The model selection experiment (4.3) either is not written accurately or feels obvious based on what’s written. Other results sections - are not written to explain to a human reader in my opinion, they seem like LLM-written parts of the paper that were never checked by a human before submitting.

RAW REVIEW

Intro

  • Table 2 referred to in the main text but it’s in the appendix, but it seems like a critical table to convey the points in the intro. Please either bring it up or explain why it is in the appendix.

Related Work

  • Table 1 needs to cite specific benchmarks to compare to, instead of grouping them and not citing any

    • It is unclear what any of the rows actually mean as they don't refer to standardized terms
    • Same with the columns - it is not explained what each of it actually means
  • The paper relies heavily on extracting structural time series features, but does not cites papers on standard feature extraction frameworks like tsfeatures [1] or catch22 [2] which are heavily used in research. It's unclear how their proposed features compare to or differ from these existing sets. They however cite papers like Kang et al (such as GRATIS), so not all relevant literature is missed.

[1] Hyndman, R., Kang, Y., Montero-Manso, P., O'Hara-Wild, M., Talagala, T., Wang, E., & Yang, Y. (2026). tsfeatures: Time Series Feature Extraction (R package version 1.2.0.0). https://pkg.robjhyndman.com/tsfeatures/ [2] Lubba, C. H., Sethi, S. S., Knaute, P., Schultz, S. R., Fulcher, B. D., & Jones, N. S. (2019). catch22: CAnonical Time-series CHaracteristics: CH Lubba et al. Data mining and knowledge discovery, 33(6), 1821-1852.

Section 3

  • Figure 2 is useful but very cryptic; I did not fully understand it
    • E.g. Family-Level Sifting, Salience coefficient - unsure what that means
  • “The second adds intervention-response metrics that measure how faithfully a forecast tracks each induced change”
    • I don’t understand what this means. "forecast tracks each induced change" - what does "forecast track" mean here - what is a forecast supposed to track? Do you mean how well a model produces a forecast of time series that has been changed that way? Further, I don't know what "intervention response metrics" means - it is not a standard term in the literature. Do you simply mean a forecasting metric on the intervened time series?
  • The structural features make sense at a high level, but I know that the low-level implementation of each can vary widely. There are no standardized/accepted definitions of each. The main text (Sec 3.1) does not provide the low-level implementation / assumptions made for each; does not point to a relevant appendix section as well.
  • “removing edge-of-distribution windows so that the refreshed set stays within the benchmark’s typical structural range”
    • I am unable to understand the rationale behind this. Ideally one would want to keep the long tail of forecasting windows or at least report them separately
  • The augmentation part of the framework does not make sense to me / it is extremely unclear from the text (Sec 3.2) what it means:
    • Do you take a single time series and generate samples that change one structural feature?
    • What guarantees that this generated time series is realistic / scores a realistic property? It would be better to simply filter out the existing time series based on their properties and produce a better “view” of the benchmark. This, in my opinion, would have been a solid contribution instead of the proposed augmentation approach.
  • Section 3.3: Inspection and Sifting: very much unclear how this is done. Both of these steps seem like non-trivial steps
  • Sentences like “Any cross-feature summary is a secondary equal-weight macro over direction-corrected feature-local ranks and does not pool raw values or define a cross-feature capability scale” are extremely cryptic (use terms not defined before in the text) and make me suspect heavy AI copy-editing of the text, which renders the text unreadable.
  • I do not have enough information to really understand the figures. The captions do not summarize / help the reader enough in order to understand them.
  • The results section is completely unreadable - the text does not summarize key results and is presented very verbosely
    • E.g. (Reference MASE and Probe MASE are rank correlated at 0.943, 0.700, 0.900, and 0.657; Appendix C).
    • “At the aggregate level the two rankings are largely consistent: the Spearman correlation between Probe-MASE rank and NRMSE rank is 0.029, 0.800, 0.900, and 0.771 on the four suites, and the refreshed set does not distort accuracy rankings”
      • It is really unclear what any of this refers to at the low level. Seems like an AI dump of the results without any understanding of what’s a significant result
  • Sec 4.4, 4.5, 4.6 are completely unreadable as well. Everything there onwards I don’t understand.
Questions for authors

Recommendations for the Authors

  • I recommend the authors carefully audit the method and results texts so it’s clear to them what it means, and they fully understand each sentence. The goal is not to simply use an AI to build a benchmark extension and get the results, but to convey them meaningfully to other readers.
  • The results section should be refreshed completely in my opinion, first introducing what measures / metrics make sense for such tasks; then gradually explaining the results. Limit the number of results you want the readers to takeaway in the main text. Fewer, well conveyed results - are much better in my opinion than dumping every low level statistic and conclusion in the main text, expecting readers to understand everything.
  • More specifically, if it is a paper where a single metric doesn’t summarize everything, then the authors should make an even better effort to convey the results and provide useful takeaways.
  • The figures and tables must make sense, standalone. This means, any jargon or term introduced in the figures must be explained in the caption. For figures/tables with results, the caption should “summarize” the results.

Questions:

  1. What are the 3-4 key takeaways that you would like readers to take from the results? These takeaways could obscure low level details, but are important for readers like me who are looking for such takeaways (instead of the correlation score between each model in each metric)
  2. Please clearly motivate the rationale behind each decision made in developing CaFE. E.g. why these features were chosen, why not others (tsfeatures)
  3. One algorithm that explains the entire procedure would really help.
2 comments
AuthorsOpenReview ↗

Official Response to Reviewer X1sw [1/2]

Thank you for the thorough review and detailed suggestions.

Q1: About the key takeaways from the results.

Section 4 now emphasizes four findings:

  1. Accuracy and response fidelity are complementary (Section 4.2): TimesFM 3.0 leads in Probe MASE on at least five of eight SFs in every suite, yet its NRMSE on SF1 exceeds 1 in every suite, whereas Linear Trend follows the strengthened trend almost exactly despite the worst Probe MASE.
  2. Per-SF evaluation improves model selection on unseen real data (Section 4.3): recommending by the relevant SF raises the Top-3 hit rate from 56.4% (aggregate leaderboard) to 61.1% (Probe MASE) and 62.1% (NRMSE).
  3. Current TSFMs make limited use of auxiliary inputs (Section 4.4): most Auxiliary-Input Gaps are near or below zero, and models that benefit more respond more faithfully to relational changes.
  4. CaFE data can also improve models (Section 4.6): joint fine-tuning of Chronos-2 reduces held-out Probe MASE by 3.80% and NRMSE by 20.40%.

These conclusions are stable under benchmark refresh (Section 4.5).

Q2: About the rationale behind each design decision.

Each decision serves a specific measurement need:

  • Eight SFs: they are representative, cover single-target, multi-target, and covariate inputs, and can instantiate and test the CaFE evaluation (Section 3.1; C8qM:W3).
  • Difference from tsfeatures and catch22: these frameworks describe observed series with statistical features, whereas an SF is a controlled change whose salience can be varied on a fixed task (Section 2; X1sw:W6).
  • Excluding weakly expressed windows: this yields References with clear evidence of the feature, so that augmentation strengthens a structure that actually exists (Section 3.2).
  • Inspection and Sifting: a sampled coefficient does not always realize the intended change, so each sample is re-measured and only complete families are admitted (Section 3.3).
  • Augmentation instead of filtering: filtering shrinks scarce data and cannot produce paired changes; removing the construction stages reduces model-selection transfer to the baseline level (vune:Q4).

Q3: About an algorithm for the entire procedure.

Algorithm 1 (Section 3.4) now traces the full procedure from References through family generation, Inspection, and Sifting to scoring. Appendices A and B give the SF constructions and checks, and Appendix B.4 walks one SF1 family through them.

R1 & W1: About the clarity of Sections 3 and 4.

We have substantially rewritten both sections and audited them sentence by sentence. Section 3 defines the key terms and adds Algorithm 1 and a worked SF1 example in Appendix B.4 (C8qM:W1). Section 4 is organized around one finding per subsection (X1sw:Q1). We removed or rewrote the quoted sentences; the specific changes are detailed below.

R2: About restructuring the results section.

We revised and followed this structure: Section 3.4 explains the metrics; Section 4 previews the five analyses; Sections 4.2–4.6 present their findings and supporting statistics. Detailed numerical tables are in Appendix C.

R3: About conveying results when no single metric summarizes everything.

Section 3.4 now states that Probe MASE, Forecast-Change NRMSE, and the Auxiliary-Input Gap measure accuracy, response fidelity, and input utility, so we report them side by side rather than as one score. Sections 4.2, 4.4, and 4.6 then show how accuracy and response complement each other on SF1, how NRMSE and the Gap relate, and how training on each metric affects the other; the takeaways are summarized in (X1sw:Q1).

R4 & W14: About standalone figures and captions.

We revised the captions to explain how to read each figure and what it tells us. Figure 2 clarifies how the stages turn a Reference into a family of augmented samples and then evaluate model forecasts. Figures 4 and 5 distinguish the rankings for forecast accuracy and response fidelity. Figure 6 explains that the radar charts show Forecast-Change NRMSE, with lower values farther out, and that models should be compared within each SF because the axes have different scales.

Figure 7 highlights the association between lower NRMSE and greater benefit from auxiliary inputs. Table 2 tests model recommendations on held-out real data and shows the advantage of selecting models by the relevant structural feature. Figure 8 shows that joint fine-tuning can improve both accuracy and response fidelity on held-out series.

W2, W4 & W5: About Tables 1 and 2 and the usefulness of core tables and figures.

Table 1 now names and cites representative benchmarks, including M4/M5, TFB, and BasicTS+, and its caption defines the four compared properties. The Introduction’s mistaken reference to Table 2 now points to Figure 3, which summarizes the eight SFs; detailed definitions remain in Appendix A, Table 4. The results are restructured as described in (X1sw:R2).

AuthorsOpenReview ↗

Official Response to Reviewer X1sw [2/2]

W3: About the model-selection experiment.

We have rewritten the protocol in Section 4.3, with details in Appendix C.3. Within each benchmark, we hold out the datasets of one source family, recommend the Top-3 models from the other families by aggregate Reference MASE or by SF-specific Probe MASE or Forecast-Change NRMSE, and count a hit when the recommendation contains the best-MASE model on held-out, unaugmented windows that exhibit the SF. The result is not trivial: SF-specific scores are computed on generated samples from other source families, yet they are tested on real data that never entered the ranking. Over 211 items, the Top-3 hit rate rises from 56.4% to 61.1% and 62.1% (vune:Q1(i)).

W6: About tsfeatures and catch22.

Section 2 now cites tsfeatures and catch22 and states the difference: such descriptors summarize an observed series, whereas an SF is a controlled change whose salience can be varied, which CaFE applies to the source task as a paired change.

W7: About Figure 2.

Figure 2's caption explains each stage and the salience coefficient λ. Section 3.2 defines the depicted sample family, and Appendix B.4 traces one SF1 family through the procedure.

W8: About the meaning of intervention-response metrics.

Intervention-response metrics do not simply recompute an ordinary forecast error on the intervened series. They compare the forecast for a generated sample with the forecast for its Reference and ask whether this forecast change matches the change we induced. Section 3.4 gives the NRMSE formula and how to read its values: 0 for an exact response, 1 for no response, and above 1 for worse than no response. Appendix B.4 gives a numerical example.

W9: About low-level implementation of each SF.

Section 3.1 now points to Appendices A.4–A.11 for each SF’s source evidence, transformation formula, parameter range, and checks, summarized in Table 4.

W10: About removing edge-of-distribution windows.

The removed windows are those in which the selected feature is only weakly expressed. Strengthening a structure that is barely present would produce an unreliable change, so excluding them yields References with clear evidence of the feature (Section 3.2). These windows remain References for the original task and for other SFs. We will report results on the excluded long-tail windows in the appendix of the revised version.

W11: About what augmentation does and whether it is realistic.

Yes, augmentation fixes one Reference and one SF and generates a family of K = 3 samples at randomly drawn salience coefficients, all sharing the same Reference and forecasting task (Section 3.2; Appendix A.2). For realism, see (C8qM:Q1); for filtering existing series instead of augmentation, see (vune:Q4) and (C8qM:W2).

W12: About how Inspection and Sifting are done.

Section 3.3 explains what each step checks, with feature-specific details in Appendix B.1 and Table 5:

  • Structural enhancement: Inspection re-extracts the structural parameters from each augmented sample to verify that the realized gain falls within the allowed range for the selected SF.
  • Source properties: it checks that the properties required by the selected transformation, such as fitted residuals, seasonal frequencies, or event support, are retained, along with observation masks and applicable nonnegativity constraints.
  • Measurable forecast-horizon change: it checks that the induced change is large enough to measure the model's response reliably.
  • Sifting: a family is admitted only if all K samples pass; otherwise it is rejected and the failed check is recorded.

Inspection uses no forecasting model, and Appendix B.4 shows each check on a concrete example.

W13: About the cryptic cross-feature summary sentence.

The sentence meant that models are ranked within each SF and these ranks are then averaged, without pooling raw values across SFs. We have removed it; Section 4.2 now reports per-SF ranks directly, and any cross-SF summary is stated explicitly as a mean rank, such as Toto 2.0's mean Probe MASE rank falling from 2.125 on Short to 5.375 on Long.

W15: About verbose results and the quoted rank-correlation sentence.

We have removed the quoted sentence. Section 4.2 now states its findings directly, centered on the complementarity of accuracy and response fidelity on SF1, and the remaining results are summarized in (X1sw:Q1).

W16: About Sections 4.4–4.6.

We have reorganized the experimental content of Sections 4.4–4.6. Each subsection now opens with the question it answers and closes with its finding. The main findings are summarized in our response to Q1 above (X1sw:Q1).

Reviewer_C8qMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes CaFE, a way to partition famous leaderboards such as GIFT-Eval and Fev Bench into 8 structured features (SF) and evaluate the state-of-the-art time series foundation models along these 8 axes. The approach proposes targeted augmentations for each SF and tests the models on these targeted augmentations. Overall, when compared to a single number, such as MASE or CRPS, CaFE provides a more rigorous framework for evaluation and understanding the failure modes of the model.

Critical strengths
  1. The breakdown of MASE into different SF-specific numbers is particularly useful in identifying the model's failure modes.

  2. The paper has tested the best-performing TSFMs, and the results seem to match the current leaderboard evals as well (TimesFM3.0 being the best performer of all the tested models).

Critical weaknesses
  1. The main ideas related to reference selection, augmentation, inspection, etc., are explained on a superficial level in the main paper, and it gets a bit difficult to understand the approach.

  2. In my understanding, the augmentations with respect to a structural feature create new samples that resemble the reference, but with slight perturbations. This does not address the first problem of fixed benchmarks completely, in my opinion. With the added structures, instead of slicing benchmarks like gift-eval and fev-bench based on individual dataset names, the authors are additionally slicing along the 8 structural features, whose existence probability ultimately depends on the fixed datasets.

  3. What is the hypothesis behind choosing the listed 8 structural features? There could be other structures as well that are prevalent in these popular benchmarks. For example, which of these structural features covers time-varying trends?

Questions for authors

Can the authors address the critical weaknesses? Additionally

  1. How does the inspection process ensure that the augmentations are realistic?

  2. Figure 11 and the associated explanation are confusing. After finetuning on CaFE generated samples, is the evaluation done on the entire gift-eval and fev bench benchmarks? Also, can we conclude that while finetuning on CaFE-generated samples, the models do well on the 8 listed structures, while losing their generalizability?

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer C8qM

Thank you for the thoughtful comments and insightful questions.

Q1: About how Inspection ensures realistic augmentations.

CaFE does not synthesize series from scratch; realism is enforced at three levels:

  • Grounded in real structure: Augmentation strengthens a component identified in a real Reference, such as the SF1 trend slope, by a bounded gain (e.g., [1.25, 3.0] for SF1; [1.25, 2.0] for SF2, SF4, and SF6–SF8) within source constraints such as nonnegativity, keeping the forecast origin, horizon, observation masks, and covariate visibility unchanged (Section 3.2; Appendix A.2).
  • Sample-level Inspection: without any forecasting model, Inspection re-extracts each sample's structural parameters and checks that the realized change matches the intended one, that source properties such as fitted residuals, seasonal frequencies, and nonnegativity are retained, and that the change is measurable in the forecast horizon (Section 3.3; Appendix B.1, Table 5).
  • Family-level Sifting: a family is admitted only if all of its samples pass Inspection.

Empirically, shortlists ranked on generated samples transfer to held-out, unaugmented real windows (Top-3 hit rate 61.1%–62.1% vs. 56.4% for the aggregate leaderboard), and removing Profiling and Inspection eliminates this advantage (vune:Q4), indicating that the samples preserve the forecasting-relevant properties of real data.

Q2: About the fine-tuning evaluation and generalizability.

The evaluation is not on the full GIFT-Eval or fev-bench. We fine-tune Chronos-2 on 11,269 CaFE Reference–augmentation pairs from 23 GIFT-Eval configurations (horizon 48), split by source series into 10,159 training pairs (1,104 series groups) and 1,110 held-out pairs (110 groups), with no overlap in series or forecast windows (Section 4.6, Figure 8; Appendix C.5). Changes on held-out series relative to the initial model:

ObjectiveProbe MASEForecast-Change NRMSE
MASE only−6.58%−4.70%
NRMSE only+11.83%−20.79%
Joint−3.80%−20.40%

Each single objective favors its own metric, whereas the joint objective improves both. Fine-tuning on CaFE samples therefore does not trade generalization for structural response; it shows that accuracy and response fidelity are distinct yet can be improved together.

W1: About superficial explanation of the pipeline.

We have rewritten Section 3 so that each stage is explained in the main text with its purpose. Section 3.1 defines a Structural Feature, the Reference, and the two SF classes; Section 3.2 describes Augmentation with concrete examples and formally defines a sample family as {Vj=Tc(R,λj)}j=1K\{V_j = T_c(R, \lambda_j)\}_{j=1}^{K}; Section 3.3 states what Inspection checks and when Sifting admits a family; and Algorithm 1 summarizes the full procedure. Appendix A gives each SF's construction, Appendix B.1 the checks and thresholds, and Appendix B.4 traces one SF1 family through all four stages with concrete numbers.

W2: About remaining dependence on fixed benchmarks.

We agree that CaFE remains anchored to the original benchmark. Our goal is not to remove this dependence but to improve structural coverage and diagnostic power on real tasks:

  • Broader structural coverage: Augmentation draws random salience coefficients that make each feature more pronounced and diverse than in the fixed test set, and a new seed regenerates fresh samples (Section 3.2).
  • Paired comparison: unlike slicing, CaFE varies one structure on a fixed task, so it can measure how a forecast responds to that structure.
  • Stability: slicing alone further shrinks data already scarce for many structural conditions, making results less stable.

Unlike generating data from scratch, this keeps the real task context, and the ablation in (vune:Q4) shows that this construction matters for transfer to real data.

W3: About the choice of the eight SFs and time-varying trends.

We chose eight representative SFs that cover the input relations a TSFM faces: single-target structure (SF1–SF5), structure shared across targets (SF6), and directed dependence on another series or a covariate (SF7–SF8). Intrinsic SFs range from persistent trend and multiple seasonalities to evolving seasonality, regime switching, and sparse intermittent events (Section 3.1). The eight SFs are not a complete catalogue; a new structure can be added by specifying how to detect, augment, inspect, and score it (Section 5).

Time-varying trends are not a separate SF in this paper. SF1 targets a persistent global trend and excludes unstable trends whose half-history slopes disagree (Appendix A.4), while time-varying behavior is covered by SF3 for seasonality and SF4 for shifts between levels. As a natural extrapolation of SF1, time-varying trends and related features can be further incorporated into the CaFE implementation in future work through the same detect, augment, inspect, and score procedure.

I thank the authors for the rebuttal. However, I still think the paper overall needs major changes.

For Q1, I understand that if the CaFE samples are constructed from real-world windows, then they can be more realistic. But, this needs to be supported empirically. There are distribution similarity metrics that can be used to show sufficient evidence for this.

AuthorsOpenReview ↗

Official Response to Reviewer C8qM (Follow-up)

Thank you for the follow-up and for agreeing that constructing samples from real-world windows makes them more realistic. We agree that this argument requires empirical support. Within the rebuttal period, we ran the distribution-similarity analysis you suggested and added it to the revised paper as Appendix C.7; we report the preliminary conclusions below, and the final version will include the complete analysis.

Q1: About empirical evidence that CaFE samples are realistic.

The analysis supports the realism of CaFE samples: each sample differs from the real window it is built from mainly in the one structure that CaFE strengthens, and its shape stays as close to real data as real windows are to each other.

We compare CaFE samples with real windows of the same task on 455 task–SF comparisons from GIFT-Eval Short, Medium, and Long and the full fev-bench. Shape refers to 25 features that describe the pattern of a series independently of its level and amplitude, such as catch22 features, autocorrelations, and spectral entropy.

SFChange in the strengthened structureChange in shapeReal-to-real distance in shapeExtra distance of CaFE samples in shape
SF1 Trend1.180.190.0840.085
SF2 Multi-Seasonality0.300.130.0970.049
SF3 Time-Varying Seasonality0.350.020.1200.008
SF4 Regime Switching0.230.020.0990.008
SF5 Intermittency0.580.010.1430.022
SF6 Common Factor0.200.040.1210.085
SF7 Cross-Series Dependence0.150.050.0930.067
SF8 Covariate Response0.170.050.0610.035
  • Change columns: how much a CaFE sample differs from its own source window, in units of the spread of that feature across real windows.
  • Real-to-real distance: the MMD² between two separate sets of real windows from the same task, which measures natural variation.
  • Extra distance: how much farther CaFE samples are from real windows than the real-to-real distance, so 0 means as close as real windows are to each other.

The results:

  • The change is where we intend it: on every SF, the strengthened structure changes at least twice as much as the shape.
  • The shape stays realistic: on every SF, the extra distance is at most about the real-to-real distance; it is smaller on SF2–SF8 and only 6%–16% of it on SF3, SF4, and SF5.
  • Strengthened values stay within the real range: on the samples of the main evaluation, 88.0% of the strengthened values lie within the range that covers 99% of real windows from the same dataset, and at least 70% on every SF, compared with 96.9% for real windows themselves.
  • SF1 by design: SF1 uses a wider gain range, [1.25, 3.0], than the [1.25, 2.0] of SF2, SF4, and SF6–SF8 (Appendix A.2), so that whether a model extrapolates or damps a strengthened trend is clearly measurable (Section 4.2). Its larger change and extra distance follow from this choice.

In short, CaFE samples stay realistic while adding the structural diversity that a fixed test set lacks: each sample keeps the character of its source window and varies one structure in a controlled way. This added diversity serves model selection on real data: rankings computed on CaFE samples are tested only on held-out, unaugmented real windows, and on SF6–SF8, Forecast-Change NRMSE raises the Top-3 hit rate from 28/40 to 33/40 (vune follow-up, item 1).

The revised paper adds this analysis as Appendix C.7, "Distributional Similarity to Real Data", which also reports the distance over all features, a comparison with real windows of matched structural strength, and 95% source-family bootstrap intervals for every entry.

Reviewer_vuneOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CaFE (CapabilityFocused Evaluation), a refreshable evaluation-extension grounded in original benchmark data that enriches static benchmarks with structural diversity and enriches single-score black-box evaluation with multi-dimensional, capability-oriented assessment. Experiments show that CaFE can achieve stable results across GIFT-Eval and FEV-Mini20; and reveal per-feature strengths and weaknesses that a single score usually hides.

Critical strengths

The most critical strength I’d say is the idea. Although previous works have established that TSFMs benchmarks have such problems and benchmark scores do not reflect real capabilities and drawbacks properly; but this work shows a working method to address those issues with developing a clever evaluation harness.

Critical weaknesses

My critical concerns are related to experimental scope and findings.

  • Firstly, the model selection experiment is related to intrinsic features (SF1-SF5) but not to relational features (SF6-SF8). Although they admit this, it is still a big gap; relational features are arguably where the TSFMs perform the best, compared to normal models; so, that is important.
  • Secondly, the validation of SF6-8 shows very small and inconsistent gains, and maybe CIs are crossing zero; making CaFE unreliable in relational capabilities.
  • Thirdly, the paper uses metrics like, Forecast-Change NRMSE and Auxiliary-Input Gap to check how models react to changes; and they often disagrees; making the situation confusing for users. Also, standard metrics like RMSE, MAE, MAPE and all are missing; which should be kept to ensure that the model performances are stable.
  • Fourthly, they claim that their framework is refreshable and stable. But, it is unclear from the current stage of experiments, as they used only 5 seeds on one specific dataset (GIFT-EVal short) and it is not enough for the claim.
Questions for authors
  1. Following the critical weakness, my questions are: i. Can you please experimentally show that CaFE works experimentally well in relational setups too? Ii. Can you also show results and performance gains in other datasets too; and how to interpret the results more reliably, if the metrics disagree? ii. Can you please strengthen the claim of being refreshable and stable experimentally?
  2. As the metric can be cheated easily by a simple linear trend model in SF1, how can we trust the same metric to accurately measure complex features?
  3. Is it unclear how exactly an user should use the per-feature metric to choose a proper model for a given task? For example, if a model is better on SD1-3 and another model is good on SF4-8, what to choose?
  4. CaFE pipeline has a lot of steps and parameters. Can you please show some comparison with some simple alternatives to justify the overhead?
  5. Some relevant research works are missing. The whole motivation is very relevant to this position paper from ICML 2026 by Syed et al. (Position: Time-Series Foundation Models Require Explicit Domain-Level Benchmarks). Can you please check and discuss such remaining works?
5 comments
AuthorsOpenReview ↗

Official Response to Reviewer vune [1/2]

Thank you for the constructive feedback and detailed suggestions.

Q1(i): About model selection on relational features.

We have extended the model-selection experiment from the five intrinsic features to all eight SFs (Section 4.3; protocol in Appendix C.3). Holding out one source family at a time, we recommend the Top-3 models from the remaining data by aggregate Reference MASE or SF-specific Probe MASE or Forecast-Change NRMSE, and count a hit when the recommendation contains the best model on held-out original windows exhibiting the feature.

SFItemsReference MASEProbe MASEForecast-Change NRMSE
SF611697
SF79727
SF820141816
SF6–SF840272930
All SFs211119 (56.4%)129 (61.1%)131 (62.1%)

On the relational features SF6–SF8, SF-specific recommendations achieve 29/40 and 30/40 hits, compared with 27/40 for the aggregate leaderboard, confirming that CaFE also provides better model recommendations in relational settings.

Q1(ii)-a: About results on other datasets.

The submission reports complete results on four suites, GIFT-Eval Short, Medium, and Long and fev-bench Mini-20 (Section 4.2; Appendix C.2), covering short to long forecast horizons. The model-selection experiment additionally uses the full fev-bench (Appendix C.3). We will further extend the per-SF capability evaluation from fev-bench Mini-20 to the full fev-bench in the revised version.

Q1(ii)-b: About interpreting the results when metrics disagree.

The metrics measure different aspects of forecasting and are complementary rather than conflicting:

  • Probe MASE measures endpoint accuracy: how close the forecast is to the ground truth.
  • Forecast-Change NRMSE measures response fidelity: whether the forecast follows a controlled structural change. For example, if the seasonal amplitude in the history is doubled, a faithful model doubles its forecast amplitude too, even if its endpoint error grows.
  • Auxiliary-Input Gap measures input utility on SF6–SF8: the increase in MASE after removing the auxiliary variables. Because an affected target often reveals the change in its own history, low MASE and NRMSE alone do not show that a model uses the other variables; the Gap tests this directly.

A model can therefore be accurate yet unresponsive to a structural change, or the reverse. SF1 on GIFT-Eval Short illustrates this: all six foundation models beat Naive on Probe MASE (2.05–2.64 vs. 4.37), yet their NRMSE of 1.18–2.23 lies above the zero-response reference of 1, whereas Linear Trend has the worst Probe MASE but the lowest NRMSE (vune:Q2). Both views carry selection-relevant information: NRMSE gives the highest overall Top-3 hit rate (62.1%), while Probe MASE is stronger on SF4–SF6 and SF8 (Section 4.3).

Q1(iii): About refreshability and stability.

We have extended the refresh experiment from five seeds on GIFT-Eval Short to ten seeds on GIFT-Eval Short, Medium, and Long (Section 4.5; Appendix C.4).

SuiteRelative SD, Probe MASERelative SD, Forecast-Change NRMSESpearman ρ across 45 seed pairs
Short0.27–0.42%0.38–0.81%0.987 ± 0.017
Medium0.18–0.26%0.76–2.15%1.000 ± 0.000
Long0.35–0.56%0.72–1.67%1.000 ± 0.000

The relative standard deviation stays below 0.57% for Probe MASE and 2.15% for Forecast-Change NRMSE. Rankings are identical across all seeds on Medium and Long; on Short, only neighboring models with nearly identical means occasionally swap, such as TimesFM 3.0 and Toto 2.0 (both 1.29 in mean Probe MASE). These results confirm that refreshing the structural perturbations preserves the model comparison.

Q2: About trusting the metric on complex features when a linear model wins on SF1.

The advantage of Linear Trend on SF1 is expected rather than a loophole: SF1 strengthens a linear trend slope, which a linear extrapolator reproduces by construction (Section 4.2). It also exposes a meaningful behavior: TSFMs tend to damp a strengthened trend toward the mean rather than extrapolate it, in line with the known strength of linear models at trend extrapolation (Zeng et al., 2023). The metric remains informative on complex features:

  • No shortcut on relational features: matching the induced change on SF6–SF8 requires exploiting cross-series or covariate information, which no simple model achieves by construction. Accordingly, the best simple baseline outperforms at most two of the six foundation models on any of SF6–SF8 in any suite, and on GIFT-Eval Short every foundation model outperforms every baseline (Figure 5; Appendix C.2).
  • Consistency across features: a model that follows changes consistently across many SFs, rather than on a single one, demonstrates genuine ability to model structural patterns.
AuthorsOpenReview ↗

Official Response to Reviewer vune [2/2]

Q3: About choosing a model when per-SF strengths conflict.

Model selection depends on the structure, inputs, and error costs of the target task, and CaFE provides a capability-oriented view of it rather than a universal rule. A user who can estimate the dominant structures of the target data can consult the corresponding per-SF results: if one model leads on SF1–SF3 and another on SF4–SF8, the dominant structures of the task decide, and the choice is confirmed on validation data. Section 4.3 validates this usage: recommending by the SF that the held-out data exhibit raises the Top-3 hit rate from 56.4% to 61.1%–62.1%. Combining multiple SFs, both metrics, and task requirements into a single recommendation strategy is our future work (Section 5).

Q4: About comparison with simpler alternatives.

We compared the full pipeline with a direct-augmentation variant that perturbs source windows without Profiling and Inspection, on the same 211 model-selection items (Appendix C.3, Table 13). Removing these stages lowers the Top-3 hit rate from 61.1% to 56.4% for Probe MASE and from 62.1% to 55.9% for Forecast-Change NRMSE, back to the level of the aggregate leaderboard (56.4%). The data-quality stages are therefore what make the per-SF scores informative: without them, a change may not correspond to a structure present in the window, and a weak or off-target change makes NRMSE measure the wrong thing.

Filtering existing windows by structure is another simple alternative, but it further shrinks data that are already scarce for many structural conditions, making results less stable, and it cannot vary one structure on a fixed task to measure a paired response. All parameters and their ranges are listed in Appendix A.2 and Table 5.

Q5: About missing related work by Syed et al.

Section 2 now cites Syed et al. and notes that domain-level benchmarks organize data by application domain, whereas CaFE targets the temporal structure of the series themselves.

W1: About model selection on relational features.

The model-selection experiment now covers SF6–SF8, where SF-specific recommendations outperform the aggregate leaderboard (vune:Q1(i)).

W2: About small and inconsistent gains on SF6–SF8.

The small gains reflect current model behavior rather than a failure of the metric:

  • Limited input utility of current TSFMs: the Auxiliary-Input Gap measures the increase in MASE after removing the designated auxiliary input. Most measured gaps are near or below zero, a pattern already present on the original benchmark tasks; only a few model–SF pairs exceed 0.1, such as TimesFM 3.0 on SF6 (0.12), Toto 2.0 on SF7 (0.44), and TimesFM 2.5 on SF8 (0.41) (Section 4.4).
  • Discriminative relational evaluation: models with larger gaps have lower Forecast-Change NRMSE in every SF6–SF8 panel of Figure 7, simple baselines rank below most foundation models on SF6–SF8 (vune:Q2), and SF-specific relational recommendations transfer to held-out real data (vune:Q1(i)).

W3: About disagreement between metrics and missing standard metrics.

Probe MASE and Forecast-Change NRMSE measure accuracy and response fidelity, respectively, so they can rank models differently without either being wrong (vune:Q1(ii)-b). The fine-tuning experiment confirms this: optimizing NRMSE alone reduces it by 20.79% while increasing Probe MASE by 11.83% (Section 4.6). Forecast-Change NRMSE and the Auxiliary-Input Gap largely agree in direction, as models with larger gaps tend to have lower NRMSE (Figure 7). They are not interchangeable, however: on SF7, Toto 2.0 has the largest Gap but not the lowest NRMSE. We therefore report them side by side rather than as one score (Section 3.4).

Regarding standard metrics, we have added Probe MAE and Probe RMSE (Section 4.2; Appendix C.2, Figures 9–10 and Tables 11–12). Because both retain the units of each dataset, we divide each model's score by that of Seasonal Naive within each configuration and SF and take the geometric mean of these ratios across configurations. Both rankings agree closely with Probe MASE, with a mean Kendall's τ across SFs of 0.88–0.92 for MAE and 0.87–0.91 for RMSE in each suite. TimesFM 3.0 again ranks first on at least five of eight SFs in every suite under both metrics, and every foundation model has lower MAE and RMSE than Seasonal Naive on all SFs of GIFT-Eval Short and fev-bench Mini-20. The accuracy conclusions are therefore stable across error metrics. MAPE is undefined or unstable near zero, which is common in the intermittent series of SF5, so we report MAE and RMSE instead.

W4: About evidence for refreshability and stability.

The refresh experiment now uses ten seeds on GIFT-Eval Short, Medium, and Long, and both scores and rankings remain stable across all three suites (vune:Q1(iii)).

Thank you for the response and new experiments. Overall, I would say -

  • Fully addressed: W3 standard metrics (MAE/RMSE added), Q5 (Syed et al. cited), and mostly W4 (10 seeds on 3 suites, but all from GIFT-Eval);
  • Partly addressed: W1/Q1(i) (SF6-8 selection, no CIs), W3 metric disagreement (no user rule), Q1(ii)-a (other datasets, full fev-bench only promised), Q2 (SF1 only), Q3 (no procedure for conflicting SFs), and Q4 (filtering baseline and cost not run); not addressed: W2 (no CIs, no positive control).

Remaining/new concerns:

  1. SF6-8 selection gains are 2 to 3 hits of 40, and Probe MASE changes by +3, -5, +4 across SF6-8. Please add CIs and a chance baseline.
  2. Most Auxiliary-Input Gaps are near zero. Please add CIs and a positive control showing the metric detects real input use.
  3. You say the target's history often reveals the change, yet that matching it needs cross-series information. Please reconcile, e.g., with a shuffled-covariate control.
  4. Please give a rule for when Probe MASE and NRMSE disagree, and run the filtering baseline from Q4.

My rating stays for now; I will revisit if these are addressed; but looking at everything (the reviews, rebuttal depth and remaining concerns, other reviewers concerns), I'll be honest - I'm bit unsure about raising score at this moment; but best wishes!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Official Response to All Reviewers

We sincerely thank all reviewers for their careful reading and constructive feedback. In the revised manuscript, substantive revisions, including new experiments, results, and method details, are marked in red, and language polishing is marked in dark blue.

AuthorsOpenReview ↗

Summary Letter to the AC and All Reviewers

Dear AC and Reviewers,

We sincerely thank you for your time and effort in handling and reviewing our paper. Your detailed comments have substantially improved both the presentation and the evidence. Below we summarize the recognized contributions, the main comments, and how the revision addresses them.

Our paper introduces CaFE, a capability-focused evaluation extension for time series foundation models (TSFMs). Grounded in original benchmark data, CaFE strengthens one Structural Feature (SF) at a time on real forecasting tasks, verifies every generated sample, and reports per-SF accuracy and response fidelity beyond a single aggregate score. On six TSFMs and four suites, it exposes capability differences that aggregate scores hide and improves model selection on held-out real data.

(1) In the initial reviews, all three reviewers recognized the motivation and idea of CaFE and its value for diagnosing models:

  • Reviewer vune regards the idea as the most critical strength, notes that the work "shows a working method to address those issues" with "a clever evaluation harness", and summarizes that the experiments "reveal per-feature strengths and weaknesses that a single score usually hides".
  • Reviewer C8qM finds the SF-specific breakdown of MASE "particularly useful in identifying the model's failure modes", considers that CaFE "provides a more rigorous framework for evaluation", and notes that the paper tests "the best-performing TSFMs" with results that "seem to match the current leaderboard evals".
  • Reviewer X1sw states that "the two key points in the motivation of the paper are strong" and that the high-level idea, "to profile benchmarks and give more dimensions of evaluation", is also strong.

(2) The reviewers' comments fall into four groups, all addressed in the revision and our responses:

  • Clarity of the method and results (C8qM:W1, X1sw:W1, X1sw:R1, X1sw:R2). We carefully revised Sections 3 and 4 sentence by sentence. Section 3 now defines every key term, Algorithm 1 traces the full procedure, and Appendix B.4 walks one SF1 family through all four stages. Each subsection of Section 4 states one finding, captions are revised for standalone reading, and Table 1 cites specific benchmarks. The four key takeaways are listed in (X1sw:Q1).
  • Experimental scope and stability (vune:W1, vune:W4, X1sw:W3). Results cover four suites (vune:Q1(ii)-a). The model-selection experiment now covers all eight SFs: over 211 held-out items, the Top-3 hit rate rises from 56.4% for the aggregate leaderboard to 61.1%–62.1% for SF-specific recommendations, with 29/40 and 30/40 hits versus 27/40 on the relational SF6–SF8 (vune:Q1(i)). With ten seeds on three suites, the relative standard deviation stays below 0.57% for Probe MASE and 2.15% for Forecast-Change NRMSE, and rankings are identical across seeds on Medium and Long (vune:Q1(iii)). Fine-tuning is now evaluated on held-out series, where the joint objective reduces Probe MASE by 3.80% and NRMSE by 20.40% (C8qM:Q2).
  • Metrics and their interpretation (vune:W2, vune:W3, X1sw:R3, X1sw:W8). Section 3.4 explains that Probe MASE, Forecast-Change NRMSE, and the Auxiliary-Input Gap measure accuracy, response fidelity, and input utility, so they are complementary and reported side by side (vune:Q1(ii)-b). Linear Trend's low NRMSE on SF1 is expected by construction, while on SF6–SF8 simple baselines rank below most TSFMs (vune:Q2). The Gap further reveals that current TSFMs make limited use of auxiliary inputs, and models that benefit more tend to respond more faithfully (vune:W2). We added Probe MAE and Probe RMSE, and both rankings agree closely with Probe MASE, with a mean Kendall's τ of 0.88–0.92 and 0.87–0.91 (vune:W3).
  • Design rationale and realism (vune:Q3, vune:Q4, C8qM:W2, C8qM:W3, X1sw:Q2, X1sw:W11, X1sw:W12). The eight SFs cover single-target structure, structure shared across targets, and directed dependence on another series or a covariate. Realism is enforced by grounding each change in real structure, sample-level Inspection, and family-level Sifting (C8qM:Q1). Compared with filtering existing windows, augmentation avoids shrinking scarce data and varies one structure on a fixed task; an ablation shows that removing Profiling and Inspection lowers the hit rate to the aggregate-leaderboard level (56.4% and 55.9%). Section 2 now discusses Syed et al., tsfeatures, and catch22 (vune:Q5, X1sw:W6).

Substantive revisions are marked in red and language polishing in dark blue.

Overall, we believe CaFE offers a practical step from single-score leaderboards toward capability-oriented evaluation of TSFMs.

If any point in the revision or our responses remains unclear, we would be very grateful if you could let us know, and we will gladly clarify or provide further analysis.

Thank you again for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.