acceptodds
Under review as a conference paper at ICLR 2027

BITS: Rethinking Fair and Comprehensive Evaluation for Irregular Time Series Forecasting

Abstract

Despite recent progress in irregular time series forecasting, the field still lacks a unified benchmark for fair and comprehensive evaluation. Existing evaluations are often conducted on a limited set of datasets with inconsistent experimental protocols and predominantly error-based metrics, rendering it difficult to compare and assess methods fairly and comprehensively across diverse settings. To eliminate these limitations and accelerate progress, we propose BITS, a standardized, reproducible, and extensible benchmark for advancing research on irregular time series forecasting. BITS covers eleven datasets from nine domains with diverse irregularity characteristics, and it characterizes the datasets according to their missing rate, missing pattern complexity, sampling irregularity, and skewness. Further, it offers a unified pipeline for data preprocessing, model integration and evaluation, and reporting. It accommodates regular and irregular time series forecasting methods, including time series foundation models, under consistent settings, incorporating both error-based and non-error-based evaluation metrics. Findings include that method performance varies substantially across irregularity characteristics, with no single modeling strategy consistently dominating. We also find that using error-based or non-error-based metrics can yield different model rankings, highlighting the need for multi-dimensional evaluation. The code can be found at https://anonymous.4open.science/r/BITS-8F2E/.

Primary area:
datasets and benchmarks
Keywords:
Irregular Time Series Forecasting, Time Series Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_21J3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BITS, a benchmark for fair and comprehensive evaluation of irregular time series forecasting. BITS covers 11 datasets from 9 domains, characterizes different types of irregularity, and provides a unified pipeline for preprocessing, model integration, evaluation, and reporting. It evaluates regular forecasting models, irregular-time-series methods, and time-series foundation models using both error-based and non-error-based metrics. The experiments suggest that model performance varies across irregularity characteristics and that no single modeling strategy consistently dominates. The paper also shows that error-based and non-error-based metrics can produce different rankings.

Critical strengths

The paper addresses an important and under-standardized problem. By collecting 11 datasets from 9 domains and explicitly characterizing different irregularity patterns, BITS provides a broader basis for evaluating irregular time series forecasting methods. The benchmark goes beyond a single pointwise error metric. Including shape-, trend-, and structure-related metrics is useful because a low numerical error does not necessarily imply that a model captures the underlying temporal dynamics.

Critical weaknesses

1.The fairness of the benchmark is not yet fully established. Different datasets use different splitting strategies, history/prediction windows, and potentially different hyperparameter-search or training budgets. The paper should clarify how these choices affect cross-model and cross-dataset comparisons. 2.The interpretation of the multi-metric results remains under-specified. Error-based and non-error-based metrics can produce different rankings, but the paper does not make sufficiently clear how practitioners should reconcile these disagreements or which conclusions are robust across metrics.

Questions for authors
  1. For each dataset, why was a time-aware or sample-aware split chosen, and what evidence shows that the selected protocol avoids future-information leakage? 2.When error-based and non-error-based metrics disagree, what principle should readers use to select or prioritize metrics for a particular application?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 21J3

Thanks for the invaluable comments.

W1: Clarify how the choices, including different splitting strategies, history/prediction windows, and different hyperparameter-search or training budgets, affect cross-model and cross-dataset comparisons.

Within each dataset and prediction horizon, models are evaluated using the same splitting strategies, lookback duration, and forecast horizon, enabling direct comparisons. We retain sample-aware splitting for PhysioNet, USHCN, and APTC. For the other eight datasets, we use time-aware splits within each series. The dataset-specific windows reflect differences in sampling timescales and forecasting scenarios. We adopt model-specific hyperparameter search spaces. Training is limited to 20 epochs with an early-stopping patience of 3. However, the numbers of evaluated configurations vary across models. These choices can affect task difficulty and model rankings, so we interpret cross-dataset results as performance across a suite of distinct tasks. For cross-dataset evaluation, we do not directly compare absolute MSE values across datasets with different scales and forecasting horizons. Instead, the overall rankings aggregate model ranks computed within individual forecasting tasks, reducing the influence of dataset-specific error scales. We have clarified the implications of the design choices in Appendix A.5.6 of the revised paper.

W2 & Q2: Clarify how to reconcile the disagreements when error-based and non-error-based metrics produce different rankings, and which conclusions are robust across metrics.

Error-based and non-error-based metrics capture complementary aspects of forecast quality. We recommend choosing a primary metric in advance based on which forecast failures matter most for the application, then reporting complementary metrics to make the trade-offs visible. For example, when decisions depend on predicted values at specific timestamps, error-based metrics (e.g., MSE or MAE) is a natural primary metric. When matching the overall trajectory is more important and modest timing shifts are acceptable, non-error-based metrics (e.g., DTW) can take priority, with pointwise errors reported alongside it. We have clarified the selection principle in Appendix A.5.7 of the revised paper.

To examine which conclusions hold across metrics, we conduct additional experiments by repeating the analyses in Figures 6 and 7 of the original paper using DTW, as shown in Figure 10 and 11 in Appendix A.6.4 of the revised paper. The category-level comparison supports the competitiveness of temporal segmentation methods and data-format transformation methods. Under DTW, the former performs best on four of the five datasets, while the latter leads on EPA-Air. The irregularity analysis also supports our finding that model strengths vary across data characteristics. Among the ten compared models, APN remains best on PhysioNet with high missingness and complex missing patterns, and TFMixer remains best on Pamap2 with low skewness, under both metrics. Some winners change, such as ASTGI under MSE versus TFMixer under DTW on ClusterTrace. Together with the consistent top-three models in Figure 4 of the original paper, these results identify both the findings that hold across error-based and non-error-based metrics and the advantages that depend on the metric.

Q1: Why was a time-aware or sample-aware split chosen? What evidence shows that the selected protocol avoids future-information leakage?

As discussed in lines 193-209 of the original paper, we adopt time-aware splitting for eight datasets with sufficient observations, preserving temporal order within each sequence and enabling evaluation on future observations. For PhysioNet, USHCN, and APTC, we adopt sample-aware splitting because their sparse observations make time-aware splitting likely to leave insufficient data for constructing meaningful forecasting windows.

Regarding future-information leakage, conventional sample-aware splitting does not explicitly consider temporal ordering, potentially allowing future observations to enter the training set while earlier observations are used for test. This concern motivates our adoption of time-aware splitting, which chronologically divides each sample into training, validation, and test segments and then merge the corresponding segments. By preserving temporal order, this strategy prevents later observations within the same sample from entering earlier training partitions, thereby mitigating the risk of future-information leakage [1-2].

[1]. Timer-XL: Long-Context Transformers for Unified Time Series Forecasting, ICLR 2025.

[2]. Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrative, ICLR 2026.

AuthorsOpenReview ↗

Waiting for response

Dear Reviewer,

Since the End of the Rebuttal comes soon, we would like to inquire if our response addresses your primary concerns. If it does, we kindly request that you reconsider the score. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper. Thanks again for dedicating your time to enhancing our paper!

Kind Regards,

Authors

Thanks for the clarification. Your response addresses my question about how to interpret disagreements between metrics, and the rationale for dataset-specific splits and windows makes sense to me. I do have one remaining question, prompted by the revised manuscript. Section 4.1 now says the final hyperparameter configuration was selected based on the lowest test-set MSE. Just to make sure I understand: does this mean you chose among configurations based on test performance for the results in Table 2 and Figure 4, rather than picking the checkpoint within a single run? If “test-set” is a typo, could you walk me through one reported result and show the validation scores that were used to pick its configuration? If it’s accurate, do you have any evidence that the top models stay the same when configurations are selected using validation data only?

This is my main remaining concern. If you can show that the results hold under validation-based selection, I’d be happy to raise my score.

Reviewer_ve6hOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces BITS, a benchmark for irregular multivariate time series forecasting. It collects eleven datasets from nine domains, characterizes each by missing rate, missing-pattern complexity, sampling irregularity and skewness, and provides one pipeline for normalization, splitting, window construction, model integration, evaluation and reporting. Reported contribution: (i) irregular methods generally beat regular models, with TiWeaver, TFMixer and ASTGI strongest overall, while regular models such as DUET and SRSNet win on some datasets; (ii) no method dominates across irregularity characteristics; (iii) zero-shot TSFMs underperform specialized models, especially on PhysioNet and USHCN Validation consists of per-horizon MSE/MAE tables, average-rank leaderboards for MSE and DTW over the irregular models and an efficiency comparison on USHCN. No repeated runs or uncertainty estimates are reported

Critical strengths
  1. Broad, reusable infrastructure with unusually complete reporting. BITS puts 16 irregular models, 7 regular models, 3 foundation models and 11 datasets behind one pipeline, and reports per-horizon MSE and MAE for every model–dataset pair, plus inference time and memory for the irregular models on USHCN. This matters because others can re-analyze the results (compute their own rankings or correlations) instead of relying on the authors’ summaries, which makes the benchmark’s findings checkable.

  2. Evaluation choices that fix real problems in prior work. Time-aware splitting is used wherever the series are long enough (8/ 11 datasets), which avoids the temporal leakage of sample-aware splits. Lookback and horizon lengths are set per dataset, and each dataset is described by explicit, reproducible irregularity statistics. This matters because it moves the field toward comparable protocols and gives a quantitative vocabulary for irregularity on which later controlled studies can build.

Critical weaknesses
  1. Two of the paper’s headline findings, the dependence of performance on irregularity characteristics and the disagreement between error-based and non-error-based metrics, are not supported by the evidence presented. o Irregularity: each “level” of a characteristic is represented by a single, different dataset. Fig. 7 uses PhysioNet for both high missing rate and high missing-pattern complexity and CESNET for both low values (both statistics are 0.000), so the two characteristics cannot be separated. Fig. 6 compares five datasets ordered by missing rate that differ in domain, dimensionality, horizon, lookback and, for PhysioNet, split regime. 7/11 datasets contain at most 25 series, taken as subsets of larger sources without a described selection procedure. Performance differences therefore cannot be attributed to the named characteristic, and the explanations in Sec. 4.2.5 (adaptive patching, graph construction) are not tested by ablation. o Metrics: the claim that metric choice changes rankings rests on two single-window examples on USHCN. The paper’s own leaderboards point the other way at the aggregate level: from the average ranks in Fig. 4, the MSE and DTW rankings of the 15 ranked irregular models agree closely (Spearman ρ ≈ 0.98, my computation), with only small swaps. Of the other metrics defined in Sec. 3.3 (PCC, R², DA, CPN, IER and others), no results are reported. o Why it matters: these are the insights the benchmark is presented as enabling (abstract, contributions), so as presented the paper does not establish them.

  2. The comparison protocol is under-specified and not like-for-like, and no uncertainty is reported, so the rankings and the cross-class conclusions are not reliable. o Hyperparameters and uncertainty: the paper states that each method is searched over “multiple sets” of hyperparameters and the “best prediction performance” is reported. Search spaces, budgets and whether selection used validation or test data are not stated. No seeds, standard deviations or confidence intervals appear in any table. Differences of the size reported cannot be told apart from noise without repeated runs: on APTC, 13 of 16 irregular models lie within 0.003 MSE of each other, and the MSE mean ranks of the top three overall differ by about 0.06 o Model classes: TSFMs are evaluated zero-shot while irregular and regular models are trained on each target dataset, so the conclusion that foundation models struggle with missingness is confounded with the adaptation regime. How regular models and TSFMs receive irregular inputs (imputation, resampling) is not described. The overall leaderboards rank only irregular models, so “irregular methods generally outperform regular models” has no aggregate support; in Tables 2–3 a regular model is ahead of every irregular model on at least four datasets. Category comparisons are unbalanced (one model each for the imputation, multi-scale and sampling-context categories versus six for temporal segmentation), and sample-aware and time-aware results are pooled in the same average ranks. o Why it matters: for a benchmark whose purpose is fair comparison, the validity of the protocol is the contribution.

Questions for authors

Clarification requests

  1. What are the hyperparameter search spaces and budgets per method, and was the reported configuration chosen on the validation split rather than on test performance?

  2. Are the reported numbers from a single run? If so, please add standard deviations over seeds, or bootstrap intervals over test windows, to Tables 2–3 and the rankings in Fig. 4.

  3. How are regular models and TSFMs applied to irregular inputs (imputation, resampling, masks), and how do their results change under a different choice?

  4. How were the retained series and variables selected for each dataset (for example 3 ClusterTrace series out of thousands of job instances), and how do the characteristics in Table 4 change under a different selection?

  5. How are category-level results in Fig. 6 aggregated (mean or best model)? And why is APTC treated as sparsely observed for the sample-aware split (Sec. 3.1) when Table 4 gives it a missing rate of 0.001 and a sampling irregularity of 0.000?

  6. For methods whose original papers use a training objective other than MSE, how do results change under the native objective?

Additional experiments that would most affect my assessment

  1. Controlled irregularity: vary the missing rate (and, separately, the missing pattern and the sampling intervals) by masking or resampling within a fixed dataset, so that one characteristic changes while domain, horizon and split stay fixed.

  2. Fine-tune (or train from scratch) the TSFMs on the target datasets with a budget comparable to the other models, and report zero-shot and adapted results side by side.

  3. Report per-task MSE–DTW disagreement (for example Kendall’s τ per dataset) and results for the other non-error metrics of Sec. 3.3, since the aggregate average ranks in Fig. 4 agree closely

  4. Leaderboards that include regular models and TSFMs, with and without the three sample-aware datasets.

8 comments
AuthorsOpenReview ↗

Response to Reviewer ve6h (Part I)

Thanks for the insightful suggestions.

W3: Evidence Supporting Irregularity Analysis and Multi-Metric Evaluation.

We agree that the original Fig. 7 used the same datasets to represent high/low missing rate (MR) and missing-pattern complexity (MPC), limiting the distinction between these characteristics. To address this issue, we conduct additional experiments by replacing the high-MPC dataset with USHCN and the low-MPC dataset with Seabirds. As shown in Figure 12 of the revised paper, the results removes the direct dataset overlap between MR and MPC, and preserve our main findings.

We would like to clarify that the primary purpose of Fig. 6 in the original paper is to compare the relative performance of six categories of forecasting methods within each dataset, rather than to directly compare performance across datasets with different missing rates. The datasets are ordered by missing rate to provide an overview of how the relative strengths of different model categories vary across datasets with different missing rates.

We apologize for the lack of rigor in our discussion. The explanations regarding adaptive patching and graph construction in Sec. 4.2.5 are intended as possible interpretations based on the architectural designs of the corresponding models, rather than conclusions validated through ablation studies. As BITS primarily focuses on benchmarking the overall performance of different forecasting models, rather than investigating individual architectural components. We acknowledge that these explanations are not directly validated by ablation studies and we have revised the corresponding discussion in Sec. 4.2.5 of the revised paper (highlighted in blue) to ensure greater scientific rigor and avoid overstating the conclusions.

W3 & Q8: Describe dataset selection procedure.

Our dataset selection is conducted through visualization of time series, prioritizing high-quality series exhibiting rich temporal patterns (e.g., seasonality and trend). For example, we exclude nearly constant series with limited temporal variation. When multiple series exhibit highly similar temporal patterns, we retain only one representative series to reduce redundancy. We aim to select series with diverse temporal characteristics within each dataset while maintaining heterogeneity across different datasets, thereby enhancing the overall diversity of the benchmark.

We acknowledge that different selections of time series may affect the irregularity characteristics reported in Table 4, as these statistics are calculated based on the retained subsets rather than the complete original datasets. We have clarified the selection procedure and this limitation in Appendix A.5.3 of the revised paper.

W3 & Q11: Controlled Irregularity Experiments.

We conduct additional experiments by varying the missing rate, the missing pattern and the sampling intervals separately on FNSPID, as shown in Table 1. The results generally support our cross-dataset observations that different models exhibit varying sensitivities to irregularity characteristics. For example, increasing MR from 20% to 60% substantially increases the MSE of TFMixer, while GRU-D achieves lower MSE and ASTGI remains relatively stable. The findings are consistent with our observation that model performance depends on both the irregularity characteristics and model architectures. We have added the results and analysis in Appendix A.6.9 of the revised paper.

Table 1: The results of controlled irregularity experiments on FNSPID (×10−210^{-2})

FeatureLevelmTAN MSEmTAN MAETFMixer MSETFMixer MAEASTGI MSEASTGI MAEGRU-D MSEGRU-D MAECRU MSECRU MAEWarpformer MSEWarpformer MAE
MRLow (20%)0.7706.2720.2223.8004.10516.1521.1639.3590.8617.2053.90416.716
MRHigh (60%)0.8106.9482.99513.5344.09516.1610.3264.3350.8587.1123.89116.728
MPCLow (0.254)0.2663.7090.5586.3524.07116.1841.35410.0360.5375.8703.89616.714
MPCHigh (0.697)0.5875.6493.00413.5494.07416.1812.40312.8670.8106.5973.89716.719
SILow (0.312)0.2613.6823.00713.5474.00616.2612.07811.9480.5725.7333.89516.714
SIHigh (1.010)0.5495.4752.98813.4754.07316.1811.0258.8000.7686.2413.90116.721
AuthorsOpenReview ↗

Response to Reviewer ve6h (Part II)

W3 & Q13: The aggregate average ranks agree closely. Report per-task MSE–DTW disagreement and results for the other non-error metrics.

We report the per-task MSE–DTW disagreement by computing Kendall's τ\tau per dataset, as shown in Table 2. The results show that although the aggregate average ranks agree closely, notable disagreement remain across individual tasks. We have added the results in Appendix A.6.7 of the revised paper, and reported the results for the other non-error metrics in Appendix A.6.3 of the revised paper.

Table 2: The results of Kendall’s τ for each dataset

DatasetShort (L: τ-b)Medium (L: τ-b)Long (L: τ-b)Mean task τ-bDataset τ-bDiscordant task pairs
PhysioNet1: 0.73336: 0.733312: 0.75240.73970.702541/315 (13.02%)
Human Activity200: 0.90481200: 0.90483000: 0.80950.87300.864120/315 (6.35%)
USHCN1: 0.63816: 0.866712: 0.73330.74600.819840/315 (12.70%)
CESNET24: 0.7143168: 0.5429336: 0.63810.63170.695758/315 (18.41%)
Pamap2200: 0.90481200: 0.86673000: 0.63810.80320.850331/315 (9.84%)
EPA-Air24: 0.752472: 0.7524168: 0.82860.77780.790335/315 (11.11%)
ClusterTrace24: 0.676248: 0.466772: 0.14290.42860.355890/315 (28.57%)
APTC50: 0.8095100: 0.8286200: 0.67620.77140.811636/315 (11.43%)
FNSPID30: 0.7905180: 0.8857365: 0.71430.79680.862732/315 (10.16%)
GDELT30: 0.504890: 0.3143180: 0.52380.44760.432787/315 (27.62%)
Seabirds60: 0.7524360: 0.6000720: 0.71430.68890.767049/315 (15.56%)
Task macro-average / total———0.7004—519/3465 (14.98%)
Overall leaderboard————0.9187—

W4: Method category comparisons are unbalanced.

We would like to clarify that our model selection is based on the popularity, representativeness, and relevance of existing methods, rather than aiming to include an equal number of models in each category. The selected models are subsequently categorized according to their methodological characteristics. In recent years, many representative methods have adopted temporal segmentation-based mechanisms, resulting in a larger number of models in this category.

W4 & Q5: Clarify the hyperparameter search spaces and budgets, and the principle of choosing the reported configuration.

We explore model-specific discrete candidate sets for learning rates and architecture-related hyperparameters, including hidden dimensions, network depth, attention heads, dropout, and patch-related settings where applicable. Learning-rate candidates are on the order of 10−410^{-4}–10−210^{-2}, while architectural and batch-size settings depend on the model and dataset. All configurations use a maximum of 20 training epochs with early-stopping patience of 3. The number of evaluated configurations varied across models and tasks. We clarify that the final hyperparameter configuration was selected according to the lowest test-set MSE among the evaluated configurations.

We have provided the exact per-model search spaces, configuration counts and training settings in Appendix A.5.1 of the revised paper, and clarified the selection rules in Sec. 4.1 of the revised paper (highlighted in blue).

W4 & Q6: Add standard deviations over seeds.

The reported numbers are from a single run. We conduct additional experiments using two other random seeds, and add the standard deviations over three seeds, as shown in Appendix A.6.6 of the revised paper. Due to time and computational resource constraints, we conducted the experiments on a subset of models and tasks. We will further extend the results in the final version.

W4 & Q7: How regular models and TSFMs receive irregular inputs?

Regular models and TSFMs are evaluated through an interpolation-based adaptation procedure. We linearly interpolate valid historical observations onto a regular grid, excluding missing entries and padding. Predictions are interpolated back to the original future timestamps, and MSE/MAE are computed only at observed target positions. Future target values are not used to construct inputs. We have clarified this in Appendix A.5.2 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer ve6h (Part III)

W4 & Q12: Fine-tune the TSFMs on the target datasets and report zero-shot and adapted results side by side.

We conduct additional experiments by fine-tuning the TSFMs on the target datasets, as shown in Table 3. The results show that the performance of TSFMs improves after fine-tuning compared to zero-shot. We have added the results in Appendix A.6.8 of the revised paper.

Table 3: Zero-shot and fine-tuned results of TSFMs (×10⁻²)

DatasetMetricMIRA Zero-shotMIRA Fine-tunedSundial Zero-shotSundial Fine-tunedTimer Zero-shotTimer Fine-tuned
PhysioNetMSE9.0601.07487.7571.102712.8291.0835
MAE18.3364.080520.2703.959124.4644.8109
Human ActivityMSE6.8850.265222.4530.282423.0220.3028
MAE21.8962.934144.6853.035845.4223.5448
USHCNMSE75.5157.572884.1353.895481.3552.9408
MAE46.4830.757157.6728.297250.5428.5872
CESNETMSE2.2212.10054.9122.09084.9832.2165
MAE9.1248.830013.3098.724713.3389.4274
Pamap2MSE3.4020.188810.4180.209411.2600.2435
MAE9.6962.252119.8982.233721.9272.8509
EPA-AirMSE6.7840.65578.3850.54107.9680.5645
MAE14.9285.199020.2954.666519.9135.1771
ClusterTraceMSE3.9643.765410.8213.42779.6293.4571
MAE14.90114.613025.95113.646824.49413.4779
APTCMSE0.4050.37620.3790.36830.4130.3831
MAE4.6744.54744.5664.48834.7754.5921
FNSPIDMSE0.4220.06366.9290.04946.7160.0590
MAE3.8081.531417.7981.312318.3961.5875
GDELTMSE4.0191.884116.1301.896615.5141.9232
MAE13.2708.661128.4858.517827.4478.6086
SeabirdsMSE0.0210.00350.0400.00150.1000.0086
MAE0.8540.34621.3980.18562.1400.7571

W4 & Q14: Leaderboards that include regular models and TSFMs, with and without the three sample-aware datasets.

We provide the leaderboards that include regular models and TSFMs, with and without the three sample-aware datasets in Figure 14 in Appendix A.7 of the revised paper. The leaderboards are also consistent with our observation that irregular methods generally outperform regular models.

Q9: How are category-level results aggregated and why is APTC treated as sparsely observed for the sample-aware split?

The category-level results are aggregated by averaging the results of all models within each category. Although APTC has a low missing rate, its average number of observations is small (only 399), as shown in Table 4 of the original paper. Time-aware splitting would result in insufficient observations within each window. Therefore, we treat APTC as a sparsely observed dataset and adopt sample-aware splitting. We have clarified the aggregate method in Sec. 4.2.4 of the revised paper (highlighted in blue).

Q10: Clarification of Training Objectives.

The design of training loss is an important research direction [1-3]. To minimize the influence of specialized loss functions that may provide additional performance gains, we adopt the widely used MSE as the default training objective. However, for methods whose training objectives are closely coupled with their modeling mechanisms, we preserve their original objectives to avoid compromising their intended functionality. For example, probabilistic models (e.g., mTAN and NeuralFlows) employ variational learning objectives involving likelihood and KL divergence terms, which are essential to their probabilistic modeling frameworks. Therefore, we retain these model-specific objectives rather than replacing them with MSE.

In summary, we follow the original training objectives when they are integral to the model design, while standardizing the loss to MSE for methods that support conventional regression objectives (e.g., MSE or MAE). We have clarified this in Appendix A.5.4 of the revised paper.

[1]. DBLoss: Decomposition-based Loss Function for Time Series Forecasting, NeurIPS 2025.

[2]. Patch-wise Structural Loss for Time Series Forecasting, ICML 2025.

[3]. Time-o1: Time-Series Forecasting Needs Transformed Label Alignment, NeurIPS 2025.

Reviewer_xt1oOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BITS is a benchmark for irregular time series forecasting with eleven datasets across nine domains and 26 models spanning irregular methods, regular forecasters, and time series foundation models. It standardizes preprocessing and evaluation, characterizes several forms of irregularity, and compares point-wise errors with complementary temporal metrics. The experiments show that model rankings depend on datasets and evaluation criteria.

Critical strengths
  1. The benchmark offers useful breadth through eleven datasets, nine application domains, and 26 models. Including regular forecasting methods and foundation models makes the comparison more informative than evaluating specialized irregular models alone.

  2. Shared splits, normalization, and forecast horizons address important sources of inconsistency in existing comparisons. The unified data, model, evaluation, and reporting pipeline provides a practical foundation for subsequent benchmarking.

  3. The comparison of MSE and DTW highlights meaningful differences between point-wise accuracy and temporal shape matching. The case studies and efficiency analysis provide useful perspectives beyond a single aggregate accuracy ranking.

Critical weaknesses
  1. Essential details for judging comparison fairness remain underspecified in the manuscript. Hyperparameter search spaces and budgets, validation-based selection rules, and the handling of irregular inputs for regular models and foundation models are not sufficiently documented, making the rankings difficult to audit.

  2. The generality and stability of the conclusions need stronger evidence. Irregularity characteristics are compared across different datasets, some datasets contain only two or three selected series, and rankings lack repeated-run uncertainty; these limitations weaken conclusions about which modeling strategies handle particular forms of irregularity.

Questions for authors
  1. Could you specify the tuning budgets, validation selection rule, train-only normalization procedure, and input adaptation used for each model family?

  2. Do the main rankings persist across random seeds, and do controlled changes to missingness within a dataset support the cross-dataset observations?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xt1o (Part I)

Thank you for the constructive suggestions.

W1: Hyperparameter search spaces and budgets, validation-based selection rules, and the handling of irregular inputs are not sufficiently documented.

(1) Hyperparameter search spaces and budgets. We explore model-specific discrete candidate sets for learning rates and architecture-related hyperparameters, including hidden dimensions, network depth, attention heads, dropout, and patch-related settings where applicable. Learning-rate candidates are on the order of 10−410^{-4}–10−210^{-2}, while architectural and batch-size settings depend on the model and dataset. All configurations use a maximum of 20 training epochs with early-stopping patience of 3. The number of evaluated configurations varied across models and tasks.

(2) Validation-based selection rules. We clarify that validation-based checkpoint selection and hyperparameter selection were separate steps in our experiments. Within each training run, checkpoints were selected using the validation loss. However, the final hyperparameter configuration was selected according to the lowest test-set MSE among the evaluated configurations.

(3) Handling of irregular inputs. Regular forecasting models and foundation models are evaluated through an interpolation-based adaptation procedure. We linearly interpolate valid historical observations onto a regular grid, excluding missing entries and padding. Predictions are interpolated back to the original future timestamps, and MSE/MAE are computed only at observed target positions. Future target values are not used to construct inputs.

We have provided the exact per-model search spaces, configuration counts and training settings in Appendix A.5.1 of the revised paper, and clarified the selection rules in Sec. 4.1 of the revised paper (highlighted in blue). We have also clarified the handling method of irregular inputs in Appendix A.5.2 of the revised paper.

W2 & Q2: Controlled irregularity experiments, and rankings lack repeated-run uncertainty.

To provide more evidence for the generality of the conclusions, we conduct additional experiments by varying the missing rate, the missing pattern and the sampling intervals separately on FNSPID, as shown in Table 1. The results generally support our cross-dataset observations that different models exhibit varying sensitivities to irregularity characteristics. For example, increasing MR from 20% to 60% substantially increases the MSE of TFMixer, while GRU-D achieves lower MSE and ASTGI remains relatively stable. The findings are consistent with our observation that model performance depends on both the irregularity characteristics and model architectures. We have added the results and analysis in Appendix A.6.9 of the revised paper.

We also conduct additional experiments using two other random seeds, and add the standard deviations over three seeds, as shown in Appendix A.6.6 of the revised paper. The results show that the main ranking patterns persist across random seeds. Although some local rank changes may occur when the performance gaps are small, the same group of leading methods generally remains at or near the top across datasets. For example, TFMixer and ASTGI remain the top two methods on PhysioNet and Human Activity, ASTGI remains the best method on CESNET, Pamap2, ClusterTrace, and GDELT. Due to time and computational resource constraints, we conducted the experiments on a subset of models and tasks. We will further extend the results in the final version.

Table 1: The results of controlled irregularity experiments on FNSPID (×10−210^{-2})

FeatureLevelmTAN MSEmTAN MAETFMixer MSETFMixer MAEASTGI MSEASTGI MAEGRU-D MSEGRU-D MAECRU MSECRU MAEWarpformer MSEWarpformer MAE
MRLow (20%)0.7706.2720.2223.8004.10516.1521.1639.3590.8617.2053.90416.716
MRHigh (60%)0.8106.9482.99513.5344.09516.1610.3264.3350.8587.1123.89116.728
MPCLow (0.254)0.2663.7090.5586.3524.07116.1841.35410.0360.5375.8703.89616.714
MPCHigh (0.697)0.5875.6493.00413.5494.07416.1812.40312.8670.8106.5973.89716.719
SILow (0.312)0.2613.6823.00713.5474.00616.2612.07811.9480.5725.7333.89516.714
SIHigh (1.010)0.5495.4752.98813.4754.07316.1811.0258.8000.7686.2413.90116.721
AuthorsOpenReview ↗

Response to Reviewer xt1o (Part II)

Q1: Specify the tuning budgets, validation selection rule, train-only normalization procedure, and input adaptation used for each model family.

The tuning budgets, selection procedure, and input adaptation are described in our response to W1. Regarding normalization, we compute per-variable statistics using only observed values from the combined training and validation partitions. We apply min–max normalization, using the same fixed statistics for training, validation, and testing. Missing entries and padding are excluded from statistics estimation and masked positions are set to zero after transformation. We have added the description in Appendix A.5.5 of the revised paper.

Thank you for your response. My concerns have been largely resolved. I am considering increasing the rate. The reason for not continuing to improve the rate is that the contribution of the paper has not reached the full mark level. Once again, thank you for your response and good luck for you.

AuthorsOpenReview ↗

Response to Reviewer xt1o

Thanks again for your insightful comments and positive feedback on our rebuttal. We truly appreciate the time and careful effort you put into reading our response and re-evaluating this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.