- Rating
- 1
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a different benchmarking view, assesing model performance by a set of capabilities instead of forecast error. It defines six tasks to represent these capabilities and tests several TSFMs on those.
Critical strengths
Benchmarking models against a set of skills the models should have, could be an interesting way of assessing the models performance without realying solely on forecasting error as it happens nowadays.
Critical weaknesses
The authors do not establish why or how these capabilites defined should be important or even correlated to the task of forecasting, which is the ultimate goal. Authors claim that there is not correlation between the performance in their tasks vs typical forecasting benchmarks (i.e. GiftEval). This claim should be supported by why, in their opinion, they should be correlated to begin with.
The authors do not explain how they evaluate these tasks they define in these models, being the core of the paper. Its not posible to understand, either replicate, these tasks' based on the paper body. The authors use the Appendix as an extension of the paper when it should be just complementary material and not in any way crucial to assess the soundness of the work.
The main contribution (the six main tasks) gets diluted between other things that do not add extra value. The dicussion on the results does not give any meaningful information since the previous part itself is not correctly and sufficienltly explained.
Too many formatting issues such as the reading flow not being one directional, there are figures without a caption (i.e. figure on page 6), demands the reader to go back and fourth between the main body and appendix to understand the fundamental parts of the work (section 3), important variables are not defined (i.e. C).
Questions for authors
Why and how are these tasks important for forecasting? Do they cover all, or most, of the capabilities a TSFM would need to do correct forecasting? i.e. are they sufficient?
Based on which rule do you decide the input length of the models? Is it related to the tasks? Why does not every model use their default context lenght?
Which parts of the paper main body could be dropped to ensure clarity and transparency in the most important part, the definition of the tasks?
Based on which decisions are the TSFMs in this work chosen? Given the wide amount of models tested, is strange to see very recent models like TimesFM3 but no Tirex2 or Chronos2 (base or small). Also given its relevance in the current literature, not adding Moirai[1,2] in favor of other models which have not gone through a reviewing process seems inconsistent (Tirex2 also applies here).
Do you plan to release this repository in a benchmarking style package such as GiftEval?
-
Woo, G., et al. (2024). Unified Training of Universal Time Series Forecasting Transformers. Proceedings of the 41st International Conference on Machine Learning (ICML)
-
Liu, X., et al. (2025). Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts. Proceedings of the 42nd International Conference on Machine Learning (ICML)
10 comments
Response to Reviewer gjR3 [2/9]
(3) Patterns and capabilities have a many-to-many relationship. This relationship directly explains why capabilities provide a higher level of abstraction.
One pattern can demand different capabilities. Periodic or seasonal sequences, for example, can be used to examine:
- Structure Induction: identifying the period and its parameters.
- History Retrieval: using relevant observations from earlier cycles.
- State Tracking: updating forecasts after changes in period, amplitude, or phase.
- Channel Modeling: using periodic information from other sequences.
- Exogenous Conditioning: responding to known periodic future inputs.
- Uncertainty Modeling: representing conditional distributions across states or forecast horizons.
A capability can also be tested through different patterns. State Tracking, for example, can be examined through changes in mean, autoregressive relationships, growth-to-plateau transitions, or frequency. These sequences have different shapes, but all require the model to update its assessment of the active dynamics from new observations.
Capability evaluation therefore organizes tasks with different shapes and origins around shared forecasting requirements. This is also why capability profiles can help explain behavior across tasks.
(4) The comparison with GIFT-Eval examines the relationship between these evaluation perspectives. We compare with GIFT-Eval to study how diagnostics of forecasting operations relate to conventional forecasting performance.
Both evaluations share a forecasting objective, so positive correlations are plausible. A model with strong overall forecasting performance may also be better at inferring structure, using historical evidence, or producing probabilistic forecasts.
Correlation asks:
Do the two measurements vary together across models?
It does not directly answer:
Does an aggregate score contain all the diagnostic information provided by the other measurement?
Aggregate forecasting performance combines tasks, conditions, and capability requirements. A model's strengths under some conditions can offset weaknesses under others. Models with similar aggregate forecasting performance can therefore exhibit different capability profiles and failure conditions.
Many capability questions also require specific controls:
- Low forecasting error does not directly establish whether distant history was used.
- Low error with joint inputs does not directly establish how much other channels contributed.
- A forecast that changes with an external input does not establish whether the direction, magnitude, and timing of that response are correct.
- Accurate point forecasts do not directly establish whether conditional intervals are calibrated.
These questions concern different evaluation targets, beyond alternative ways of ranking the same models.
(5) Correlation results support the argument after the evaluation perspectives have been explained. The correlations are best presented after the conceptual distinction, as evidence about the empirical relationship between the levels of evaluation.
We first explain the progression from datasets and patterns to forecasting operations. We then explain why these operations, which serve forecasting, should relate to aggregate forecasting performance. Finally, we show how differences between capabilities and failures under controlled conditions reveal information that aggregate rankings do not directly describe.
This organization lets the correlations support and explain the argument while keeping the conceptual contribution clear.
Our central question is therefore:
Given a model's overall forecasting performance, we still need to understand how it uses predictive information and under which requirements that use breaks down.
The value of capability evaluation does not depend on observing zero correlation or ranking reversals. It comes from the specificity of the questions being measured and the further analyses these diagnostics support.
Response to Reviewer gjR3 [3/9]
(6) Aggregate error, gains from additional information, and responses to conditions are distinct quantities. Our discussion of Channel Modeling and GIFT-Eval illustrates this distinction.
One evaluation may measure absolute forecasting error with joint inputs. Another may measure the gain from adding auxiliary channels while holding the target history and future fixed.
Both concern forecasting, but their measurement targets differ:
- The first measures final forecasting performance.
- The second measures how much additional information is converted into forecasting gains.
Similarly, Exogenous Conditioning can test whether a model responds correctly to different future conditions with the history fixed. It requires recovering the effect of the changed conditions, beyond achieving low average error.
The relationship between these scores and aggregate rankings should therefore be interpreted in light of their specific measurement targets. Capability diagnostics extend “how well does the model forecast?” to “which information contributes to the forecast, and which available information is not used effectively?”
We will revise the motivation as follows:
“We may not have made the distinction between evaluation perspectives sufficiently clear. Our contribution advances evaluation from the dataset level, through the pattern level, to the capability level. Dataset-level evaluation asks where a model forecasts well; pattern-level evaluation asks which temporal structures it handles well; capability-level evaluation asks which predictive operations it can perform and under what conditions those operations remain reliable.
This distinction matters because a pattern does not uniquely specify the operation supporting successful prediction. Accurate forecasts of seasonal sequences may rely on familiar priors, relevant historical observations, or relationships inferred from the current context. Conversely, the same capability, such as state tracking, can be examined through changes in means, trends, frequencies, or autoregressive dynamics. Capabilities therefore provide an abstraction for organizing forecasting behavior across patterns and datasets.
The comparison with GIFT-Eval examines the relationship between these perspectives. Positive correlations are consistent with their shared forecasting objective. Aggregate rankings, however, do not directly establish whether relevant history is used, additional channels provide a benefit, future conditions elicit the correct response, or predictive intervals are calibrated. Our controlled diagnostics make these questions explicit. We will revise the motivation to foreground this conceptual progression and present the correlations as evidence about its relationship to aggregate forecasting performance.”
Response to Reviewer gjR3 [1/9]
We thank the reviewer for recognizing the value of evaluating models through the capabilities required for forecasting. We take the comments on presentation and organization seriously: the original manuscript did not sufficiently explain the logic behind the capability taxonomy, and too many key task details appeared only in the appendix. We will rewrite the relevant sections so that the main text provides a self-contained account of the motivation, task definitions, and evaluation procedures.
Response: We thank the reviewer for raising the relationship between capability evaluation and forecasting performance. We may not have made the evaluation perspective sufficiently clear. Explaining this perspective requires distinguishing three levels: datasets, patterns, and capabilities.
(1) Our central contribution advances the level of abstraction in evaluation. We organize this progression as:
Dataset-level→Pattern-level→Capability-level
GIFT-Eval compares forecasting performance across datasets, domains, frequencies, and horizons. TIME advances to pattern-level evaluation by relating performance to structural time-series features. These approaches provide the context for our conceptual progression. GIFT-Eval, TIME
SortingHat makes the forecasting capability a model exhibits the unit of analysis: whether it can infer temporal structure, retrieve historical information, track changing states, combine information across channels, respond to external conditions, and represent uncertainty.
The key distinction is that patterns describe the structure present in a sequence, while capabilities describe the operations a model can perform.
Capabilities thus provide a level of abstraction for organizing forecasting behavior across datasets and patterns. Patterns serve as test settings for capabilities, while capabilities provide the basis for explaining forecasting behavior and organizing diagnostics.
(2) Pattern-level performance leaves capability-level questions unresolved. Consider two models that both accurately forecast a periodic sequence. This shows that they perform well on that periodic task, but leaves several questions open:
These sources of success and forecasting requirements may produce similar errors on ordinary tasks, yet lead to different responses when task conditions change.
For example:
“Handling periodic patterns well” therefore needs to be examined in terms of which information the model uses, which operations it performs, and under what conditions it fails.
These are sources of success that can be investigated through behavioral diagnostics. Claims about a particular internal mechanism require corresponding evidence.