- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a time series causal discovery method for linear first-order dynamics where samples are missing completely at random to recover lagged and instantaneous causal relationships. The core idea of ITSY, the proposed method, is to combine the existing algorithm DYNOTEARS with an imputation mechanism via a reconstruction loss that are both optimised end-to-end. ITSY is compared to existing methods on synthetic and semi-synthetic data.
While the problem itself (causal discovery from irregular time series) is important and worth pursuing, the proposed method is very incremental and lacks strong guarantees. This, combined with critical experimental issues, lead me to recommend a clear rejection.
Critical strengths
Problem. Time series causal discovery from irregular data is an important and understudied problem worth pursuing to improve the performance of the methods on real world data which may involve irregular samples.
Motivation. The main motivation in Section 4 and Table 1 shows importantly the core issue, which is how data irregularities affect standard causal discovery approaches.
Critical weaknesses
This is a simple and largely unsupported idea. The proposed method is essentially DYNOTEARS plus model-based imputation. The theory behind it uses existing results and lacks guarantees that ITSY recovers the true graph under missingness.
The experimental work has critical gaps. Some simple alternatives as baselines are missing (impute then DYNOTEARS; drop missing indexes and run plain DYNOTEARS). Synthetic data seems to be not standardised, which is a critical design flaw as it may favour NOTEARS-like methods (see [1]), including DYNOTEARS. Figure 3 results do not show error bars despite 50 random runs performed. It is also unclear how irregular data was provided to the baselines (padded?), and the choice of this on methods’ performance.
References
[1] Reisach, A., Seiler, C., & Weichwald, S. (2021). Beware of the simulated dag! causal discovery benchmarks may be easy to game. Advances in Neural Information Processing Systems, 34, 27772-27784.
Questions for authors
- Can you provide guarantees that ITSY recovers a true causal graph under irregular samples?
- Exactly what input the baselines received? Was it zero-padded data? Or something else? Does the choice affect how baselines perform?
- Does ITSY take advantage of non-standardised data? Will standardising the data harm its performance?
- How does ITSY compare to naïve baselines, such as impute+DYNOTEARS, or DYNOTEARS on data with dropped missing indexes?