- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper conducts an empirical analysis of missing time-series completion in spatiotemporal datasets. It investigates whether data completion can improve forecasting performance or whether models that natively handle missingness, such as Mask-DCRNN, can already perform the task effectively.
Critical strengths
- Principled evaluation methodology: Evaluates methods under different types of missingness.
- Multiple datasets and contexts: Tests the approaches across several datasets and settings.
- Honest reporting: Transparently reports the results and limitations of the empirical study.
- Baseline comparisons: Compares the proposed approach against several baselines.
Critical weaknesses
Presentation and Clarity:
- Abstract clarity: The abstract introduces many terms that are difficult to understand without reading the paper first, making it unnecessarily difficult to follow.
- Motivation for graph completion: The introduction describes graph completion as a “natural response.” However, some of the experimented datasets use artificially constructed correlation graphs, making this assumption less obvious. It is also unclear whether the relationships captured by these graphs are necessary, or whether a model could learn them implicitly.
- Output dimensionality (Section 3): (\hat{Y}) is defined over (P) and (N), without a feature dimension. It is unclear why the model does not forecast each of the (F) input channels, or whether it predicts only a single target variable per node.
Content and Contribution:
- Ambiguous contribution: The paper's main contribution is somewhat unclear. Is the goal to demonstrate that Mask-DCRNN effectively exploits missingness information, or to introduce RESOLVE? The paper acknowledges that the completion method is established and frames the work as an empirical study, but this does not by itself establish the significance of the findings.
- Unclear takeaways: even the conclusion that Mask-DCRNN handles missing values effectively is not consistently supported across all datasets. This makes the overall takeaways less clear. The paper could benefit from more discussion of the characteristics and dynamics of each dataset, potentially identifying patterns that explain why different baselines perform better in different contexts.
Questions for authors
What is the justification for the chosen 2% performance threshold?