- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces BITS, a benchmark for fair and comprehensive evaluation of irregular time series forecasting. BITS covers 11 datasets from 9 domains, characterizes different types of irregularity, and provides a unified pipeline for preprocessing, model integration, evaluation, and reporting. It evaluates regular forecasting models, irregular-time-series methods, and time-series foundation models using both error-based and non-error-based metrics. The experiments suggest that model performance varies across irregularity characteristics and that no single modeling strategy consistently dominates. The paper also shows that error-based and non-error-based metrics can produce different rankings.
Critical strengths
The paper addresses an important and under-standardized problem. By collecting 11 datasets from 9 domains and explicitly characterizing different irregularity patterns, BITS provides a broader basis for evaluating irregular time series forecasting methods. The benchmark goes beyond a single pointwise error metric. Including shape-, trend-, and structure-related metrics is useful because a low numerical error does not necessarily imply that a model captures the underlying temporal dynamics.
Critical weaknesses
1.The fairness of the benchmark is not yet fully established. Different datasets use different splitting strategies, history/prediction windows, and potentially different hyperparameter-search or training budgets. The paper should clarify how these choices affect cross-model and cross-dataset comparisons. 2.The interpretation of the multi-metric results remains under-specified. Error-based and non-error-based metrics can produce different rankings, but the paper does not make sufficiently clear how practitioners should reconcile these disagreements or which conclusions are robust across metrics.
Questions for authors
- For each dataset, why was a time-aware or sample-aware split chosen, and what evidence shows that the selected protocol avoids future-information leakage? 2.When error-based and non-error-based metrics disagree, what principle should readers use to select or prioritize metrics for a particular application?
8 comments
Waiting for response
Dear Reviewer,
Since the End of the Rebuttal comes soon, we would like to inquire if our response addresses your primary concerns. If it does, we kindly request that you reconsider the score. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper. Thanks again for dedicating your time to enhancing our paper!
Kind Regards,
Authors
Thanks for the clarification. Your response addresses my question about how to interpret disagreements between metrics, and the rationale for dataset-specific splits and windows makes sense to me. I do have one remaining question, prompted by the revised manuscript. Section 4.1 now says the final hyperparameter configuration was selected based on the lowest test-set MSE. Just to make sure I understand: does this mean you chose among configurations based on test performance for the results in Table 2 and Figure 4, rather than picking the checkpoint within a single run? If “test-set” is a typo, could you walk me through one reported result and show the validation scores that were used to pick its configuration? If it’s accurate, do you have any evidence that the top models stay the same when configurations are selected using validation data only?
This is my main remaining concern. If you can show that the results hold under validation-based selection, I’d be happy to raise my score.
Response to Reviewer 21J3
Thanks for the invaluable comments.
W1: Clarify how the choices, including different splitting strategies, history/prediction windows, and different hyperparameter-search or training budgets, affect cross-model and cross-dataset comparisons.
Within each dataset and prediction horizon, models are evaluated using the same splitting strategies, lookback duration, and forecast horizon, enabling direct comparisons. We retain sample-aware splitting for PhysioNet, USHCN, and APTC. For the other eight datasets, we use time-aware splits within each series. The dataset-specific windows reflect differences in sampling timescales and forecasting scenarios. We adopt model-specific hyperparameter search spaces. Training is limited to 20 epochs with an early-stopping patience of 3. However, the numbers of evaluated configurations vary across models. These choices can affect task difficulty and model rankings, so we interpret cross-dataset results as performance across a suite of distinct tasks. For cross-dataset evaluation, we do not directly compare absolute MSE values across datasets with different scales and forecasting horizons. Instead, the overall rankings aggregate model ranks computed within individual forecasting tasks, reducing the influence of dataset-specific error scales. We have clarified the implications of the design choices in Appendix A.5.6 of the revised paper.
W2 & Q2: Clarify how to reconcile the disagreements when error-based and non-error-based metrics produce different rankings, and which conclusions are robust across metrics.
Error-based and non-error-based metrics capture complementary aspects of forecast quality. We recommend choosing a primary metric in advance based on which forecast failures matter most for the application, then reporting complementary metrics to make the trade-offs visible. For example, when decisions depend on predicted values at specific timestamps, error-based metrics (e.g., MSE or MAE) is a natural primary metric. When matching the overall trajectory is more important and modest timing shifts are acceptable, non-error-based metrics (e.g., DTW) can take priority, with pointwise errors reported alongside it. We have clarified the selection principle in Appendix A.5.7 of the revised paper.
To examine which conclusions hold across metrics, we conduct additional experiments by repeating the analyses in Figures 6 and 7 of the original paper using DTW, as shown in Figure 10 and 11 in Appendix A.6.4 of the revised paper. The category-level comparison supports the competitiveness of temporal segmentation methods and data-format transformation methods. Under DTW, the former performs best on four of the five datasets, while the latter leads on EPA-Air. The irregularity analysis also supports our finding that model strengths vary across data characteristics. Among the ten compared models, APN remains best on PhysioNet with high missingness and complex missing patterns, and TFMixer remains best on Pamap2 with low skewness, under both metrics. Some winners change, such as ASTGI under MSE versus TFMixer under DTW on ClusterTrace. Together with the consistent top-three models in Figure 4 of the original paper, these results identify both the findings that hold across error-based and non-error-based metrics and the advantages that depend on the metric.
Q1: Why was a time-aware or sample-aware split chosen? What evidence shows that the selected protocol avoids future-information leakage?
As discussed in lines 193-209 of the original paper, we adopt time-aware splitting for eight datasets with sufficient observations, preserving temporal order within each sequence and enabling evaluation on future observations. For PhysioNet, USHCN, and APTC, we adopt sample-aware splitting because their sparse observations make time-aware splitting likely to leave insufficient data for constructing meaningful forecasting windows.
Regarding future-information leakage, conventional sample-aware splitting does not explicitly consider temporal ordering, potentially allowing future observations to enter the training set while earlier observations are used for test. This concern motivates our adoption of time-aware splitting, which chronologically divides each sample into training, validation, and test segments and then merge the corresponding segments. By preserving temporal order, this strategy prevents later observations within the same sample from entering earlier training partitions, thereby mitigating the risk of future-information leakage [1-2].
[1]. Timer-XL: Long-Context Transformers for Unified Time Series Forecasting, ICLR 2025.
[2]. Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrative, ICLR 2026.