- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents TSFMV-Bench, a benchmark for comparing impute-then-forecast pipelines and end-to-end methods for time series forecasting with missing values. It evaluates multiple methods across three datasets, seven missingness scenarios, and three missing rates. The results show that the performance gap between the two approaches varies across datasets, missingness patterns, and missing rates.
Critical strengths
S1. The paper studies a practical and relevant problem: whether to impute missing values before forecasting or forecast directly from incomplete data. S2. The benchmark compares multiple methods across different datasets, missingness patterns, and missing rates, providing a useful basis for evaluation.
Critical weaknesses
W1. The authors construct two-stage baselines primarily by combining different imputers and forecasting backbones, without sufficiently comparing representative existing impute-then-forecast methods or explaining the rationale for this modular selection. Since the groups also differ in architecture, imputation method, and training supervision, the results primarily support comparisons among the selected methods rather than attributing performance differences to the paradigms themselves. W2. The notation is inconsistent. For example, the paper defines X=[x_{t,n}] but subsequently uses z_{t,n} without explaining the relationship between them. W3. The categorization does not sufficiently account for other missing-data forecasting strategies. For example, TOI-VSF [1] proposes task-oriented imputation through joint learning, explicitly imputing missing data while allowing the forecasting objective to guide the imputation module. W4. Some citations do not match the corresponding claims. W5. The discussion in Section 5.3 is inconsistent with Figure 6. W6. There are some typos in the paper. For example, “STFMV-Bench” should probably be “TSFMV-Bench.”
[1] Hao et al., “Is Precise Recovery Necessary? A Task-Oriented Imputation Approach for Time Series Forecasting on Variable Subset,” IEEE Transactions on Knowledge and Data Engineering, 2025.
Questions for authors
See above
3 comments
W2, W4 & W6: Notation, Citation Accuracy, and Presentation Corrections
We thank the reviewer for identifying these presentation and clarity issues. We will carefully revise the manuscript to improve notation consistency, citation accuracy, and overall presentation, as detailed below.
-
Notation consistency (W2). The notation (Z) was used for the underlying reference time series in the missingness formulation, while (X) was used for the time-series observations in the forecasting formulation, without explicitly clarifying their relationship. We will unify the notation by using (X) for the reference time series and (X^{\mathrm{obs}}) for the incomplete observations, and revise the corresponding definitions, equations, and figures throughout the manuscript.
-
Citation accuracy (W4). We will carefully cross-check all citations against their corresponding claims and correct any inaccuracies. We would also appreciate it if the reviewer could point out specific instances of mismatched citations, which would help us address these concerns more precisely.
-
Typographical errors (W6). Thank you for pointing out this typo. We will correct “STFMV-Bench” to “TSFMV-Bench” and thoroughly proofread the manuscript to eliminate similar errors.
We appreciate the reviewer's careful reading and constructive suggestions, which will help improve the clarity and accuracy of our manuscript.
W5 Clarification of the consistency between Section 5.3 and Figure 6
In the paper, Section 5.3 investigates three factors associated with the relative performance of E2E and ITF: (1) cross-variable dependency structures and paradigm preferences (Figures 6(a)–(b)), (2) observation availability under different missingness patterns (Figure 6(c)), and (3) the influence of reconstruction errors near the forecasting boundary (Figure 6(d)). We will further refine Section 5.3 to explicitly discuss this contrast and clarify how each subplot supports the corresponding analysis.
Thanks for the clarification. The additional results show that the choice of imputer affects the downstream forecasting performance. However, my original concern remains. Since the ITF and E2E models differ in architecture and in whether they use imputed targets during training, it is unclear how much of the performance gap can be attributed to the training paradigm itself.
Paradigm Definition, Baseline Selection, and Evaluation Scope
W1 & W3
Regarding concerns about the representativeness of our baselines and the categorization of missing-data forecasting methods. We would like to clarify our paradigm definitions, the rationale for modular baseline selection, and the scope of our conclusions, as follows.
Clarification of paradigm definitions. Our benchmark distinguishes E2E and ITF primarily by whether imputation and forecasting are optimized independently or jointly. Specifically, ITF independently trains an imputation model before training a downstream forecasting model, whereas E2E directly optimizes forecasting from incomplete observations, potentially incorporating auxiliary reconstruction objectives. We acknowledge that task-oriented joint imputation methods, such as TOI-VSF, represent an important strategy that combines explicit imputation with forecasting-guided optimization. However, TOI-VSF primarily addresses variable-subset forecasting (i.e., complete data during the training phase and subset variable-missing during testing phase) and differs from the independently optimized ITF pipelines examined in our benchmark. We will clarify this distinction and discuss such methods in the revised manuscript.
Rationale for modular ITF baseline selection. Our modular design is intentional and reflects the standard ITF workflow. We select Mean and ImputeFormer to represent statistical and deep spatiotemporal imputation, respectively, and STID (MLP), MTGNN (CNN-GNN), and PatchTST (Transformer) to cover different forecasting architectures and cross-variable modeling strategies. Their combinations allow us to assess ITF performance across multiple implementations instead of relying on a single pipeline. To further strengthen the diversity of imputation methods, we additionally evaluated SAITS [1] combined with STID and MTGNN, providing further evidence on how imputer selection affects downstream forecasting performance. Specifically, it shows that better imputation accuracy does not necessarily lead to better forecasting accuracy. Under AQI MNAR at a 75% missing rate, repairing recent missing values reduced forecasting MAE by 2.29%, compared with 0.57% for count-matched random repairs, despite similar aggregate reconstruction errors. Moreover, the SAITS-trained predictor retained its advantage when evaluated on IF-imputed histories, suggesting that the fitted predictor also contributes to the observed performance difference.
We will refine the paradigm definitions, explain our baseline selection criteria, and clarify the scope of our conclusions in the revised manuscript.
[1] Wenjie Du, et al. SAITS: Self-attention-based imputation for time series. Expert Systems with Applications. 2023