acceptodds
Withdrawn from ICLR 2027

Fidel-TS: A High-Fidelity Multimodal Benchmark for Time Series Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The evaluation of time series forecasting models is hindered by a critical lack of high-quality benchmarks, leading to a potential illusion of progress. Existing datasets suffer from issues ranging from pre-training data contamination in the age of LLMs to the temporal and description leakage prevalent in early multimodal designs. To address this, we formalize the core principles of high-fidelity benchmarking, focusing on data sourcing integrity, leak-free and causally sound design, and structural clarity. We introduce Fidel-TS, a new large-scale benchmark built from the ground up on these principles by sourcing data from live APIs. Our experiments reveal the flaws of the previous benchmarks and the biases in model evaluation, providing new insights into multiple existing forecasting models and LLMs across various evaluation tasks.

Primary area:
datasets and benchmarks
Keywords:
Multimodal, Time Series Forecasting, High-fidelity, Benchmark, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6mPkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new time series forecasting benchmark, which is curated with API data streams. Exogenous textual information is also included for multimodal forecasting. In total, six new datasets are included in the benchmark. The paper evaluates multiple methods from diverse categories (train-from-scratch time series models, TSFMs, LLMs, multimodal forecasting models). Based on the evaluation results on these benchmarks, the paper provides several insights to a series of research questions.

Critical strengths

As a benchmark paper, the major contribution is the establishment of the newly curated datasets, especially with textual information. It is always helpful to introduce more novel datasets into the time series field.

Critical weaknesses

As a benchmark paper, it lacks an in-depth and up-to-date literature review and model comparison. The authors claim that the existing TSF benchmarks are small-scaled and theirs is large-scaled. However, there are multiple prevalent large-scaled TSF benchmarks like GIFT-Eval, fev-bench, TIME,etc. The authors do not mention or discuss these works. It also lacks a comprehensive discussion in Section 2.2 Multi-modal benchmarks for time series forecasting. Only one work is mentioned (TimeMMD), while many related works like Context is Key, MoTime[1], FinTexTS[2], FinMultiTime[3] are ignored. Regarding models, the latest time series models they used are from 2025, and the latest multimodal forecasting models are from 2024. It is highly undesirable for a 2027 paper submission.

The evaluation setup is also problematic. One conclusion the authors claim is the failure of TSFM zero-shot forecasting on their benchmark. However, they only use a fixed small context window (e.g. 360) for TSFMs, which is unsuitable. These models are supposed to use much longer context length to provide a good forecast. And context window length should be treated as a hyperparameter, as each model has its favoured context range. This evaluation protocol is unfair for TSFMs, making the conclusion also over-claimed and unsound.

[1] Motime: A dataset suite for multimodal time series forecasting, 2025 [2] Fintexts: Financial text-paired time-series dataset via semantic-based and multi-level pairing, KDD 2026 [3] Finmultitime: A four-modal bilingual dataset for financial time-series analysis, 2025

  • the references of well-known papers are not listed.
Questions for authors

What are the specific licenses of the API-curated datasets? I understand the data is publicly accessible. But it is necessary to list all the links and licenses for these data sources, indicating if they are allowed to redistribute.

What is the training protocol for the train-from-scratch time series models? How the hyperparameters are selected for them.

Can you show some full examples of the textual data samples?

Reviewer_gLp9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript introduces Fidel-TS, a multi-modal benchmark for time series forecasting that emphasizes data provenance, leakage prevention, and a clear distinction between forecasting subjects and variables. By evaluating uni-modal and multi-modal forecasting models alongside LLMs, it examines predictive performance, cross-subject generalization, and the benefits of textual information.

Critical strengths
  1. Authors address practical and important issues in forecasting evaluation.
  2. The experiments cover a relatively broad range of evaluation settings.
Critical weaknesses
  1. The model comparisons involve confounding factors. For example, FIATS uses textual information covering the prediction window, whereas GPT4MTS uses historical text.
  2. The reported sampling intervals are inconsistent. Table 1 on page 5 lists 15 minutes for GREG and 10 minutes for JAP, whereas Table 9 on page 21 lists 10 minutes for GREG and 15 minutes for JAP.
Questions for authors
  1. How do the authors ensure that textual information covering the prediction window is already available at the forecast origin?
  2. Does FIATS retain its performance advantage over other models when all models are provided with the same weather or control information?
Reviewer_iD2jOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes that existing time-series benchmarks lack fidelity, defined as low contamination, no temporal leakage, and clear demarcation of data sources. The authors propose a new benchmark that meets these standards through API data streams, controlled textual sources, and a division of numerical objectives. The authors evaluate unimodal, multimodal, and large-language-model-based time-series models to support their claim.

Critical strengths

None

Critical weaknesses

1. The paper's claim is overstated and no longer holds with recent time-series advancements.

The authors summarized unimodal existing time-series benchmarks to be in small scale, out-of-dated, but all citing older benchmarks such as M4 and Autoformer datasets, missing the more recent ones such as Gift-Eval, Monash, and the datasets used for TSFM pretraining, which already addressed scale and recency issues.

Also, the authors claimed first-generation benchmarks are with temporal leakage, but the cited Time-MMD benchmarks has already provided divisions among "fact" and "predictions" as a solution.

2. The dataset establishment and experiments cannot defend their claims well.

First, using an API does not guarantee elimination of contamination, and the authors seem to be using publicly available sources rather than the claimed private sources (for example, NYC speed can be accessed by the NYC open data platform). Even if they're private, the reproducibility will then be problematic. And the authors used the Jena database, which is the same source as Autoformer Weather, which has been criticized by them in Section 2.

Second, the text contains weather forecasts, which is a predictive context and thus cannot eliminate contamination since it messes up model forecasting and human prior (which Time-MMD already mentioned). Furthermore, in some experiments, the authors are explicitly introducing such leakage (e.g., Table 5, FIATS sees future texts).

Finally, the experiments cannot actually show if the new benchmark is with fidelity while previous ones are not. The authors are more showing performances of models on the benchmark, but not defending that their benchmark is addressing the issues where they raised in introduction.

Both points are strongly determining my recommendation.

Questions for authors

Please see the weaknesses, and these additional comments:

  1. Could the authors clarify links and APIs they used to fetch each database? If it's a private database, the authors should clarify it cleanly.

  2. Why FIATS uses future texts but GPT4MTS is restricted to past text? And why is Table 5 training the model using the union of all training sets? Furthermore, C.4 mentioned "Specifically mentioned, for all domain-specific models, both unimodal and multimodal, training must be conducted on each dataset individually." This seems to be in conflict with table 5 caption.

  3. Are TSFMs behaving better or worse under different horizons? The authors claimed TSFM is failing in forecasting, but it could also be from insufficient input horizons (for example, Chronos performs better with 512 or longer context length, as shown in the original paper)

  4. Some inconsistencies occur across the paper. For example, Table 1 GREG is 15min freq while JAP is 10 min; yet table 9 seems JAP is 15min while GREG is 10min

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.