- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a new time series forecasting benchmark, which is curated with API data streams. Exogenous textual information is also included for multimodal forecasting. In total, six new datasets are included in the benchmark. The paper evaluates multiple methods from diverse categories (train-from-scratch time series models, TSFMs, LLMs, multimodal forecasting models). Based on the evaluation results on these benchmarks, the paper provides several insights to a series of research questions.
Critical strengths
As a benchmark paper, the major contribution is the establishment of the newly curated datasets, especially with textual information. It is always helpful to introduce more novel datasets into the time series field.
Critical weaknesses
As a benchmark paper, it lacks an in-depth and up-to-date literature review and model comparison. The authors claim that the existing TSF benchmarks are small-scaled and theirs is large-scaled. However, there are multiple prevalent large-scaled TSF benchmarks like GIFT-Eval, fev-bench, TIME,etc. The authors do not mention or discuss these works. It also lacks a comprehensive discussion in Section 2.2 Multi-modal benchmarks for time series forecasting. Only one work is mentioned (TimeMMD), while many related works like Context is Key, MoTime[1], FinTexTS[2], FinMultiTime[3] are ignored. Regarding models, the latest time series models they used are from 2025, and the latest multimodal forecasting models are from 2024. It is highly undesirable for a 2027 paper submission.
The evaluation setup is also problematic. One conclusion the authors claim is the failure of TSFM zero-shot forecasting on their benchmark. However, they only use a fixed small context window (e.g. 360) for TSFMs, which is unsuitable. These models are supposed to use much longer context length to provide a good forecast. And context window length should be treated as a hyperparameter, as each model has its favoured context range. This evaluation protocol is unfair for TSFMs, making the conclusion also over-claimed and unsound.
[1] Motime: A dataset suite for multimodal time series forecasting, 2025 [2] Fintexts: Financial text-paired time-series dataset via semantic-based and multi-level pairing, KDD 2026 [3] Finmultitime: A four-modal bilingual dataset for financial time-series analysis, 2025
- the references of well-known papers are not listed.
Questions for authors
What are the specific licenses of the API-curated datasets? I understand the data is publicly accessible. But it is necessary to list all the links and licenses for these data sources, indicating if they are allowed to redistribute.
What is the training protocol for the train-from-scratch time series models? How the hyperparameters are selected for them.
Can you show some full examples of the textual data samples?