- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to train causal normalizing flows directly on incomplete data. Given a known causal graph and no unobserved confounding, MissCNF maximizes the likelihood of the observed entries and uses Monte Carlo integration only for missing ancestral variables. It also introduces causal-family positivity, under which the full joint distribution is identifiable under MAR even without complete records. Experiments cover eight synthetic causal models and three missingness mechanisms. MissCNF achieves the lowest KL divergence in 23 of 24 nonlinear MCAR/MAR settings and the lowest counterfactual error in 20. MICE generally performs best on linear models.
Critical strengths
- Causal-family positivity establishes when the joint distribution can be recovered without complete records, under a weaker condition than full-pattern positivity. This is the main theoretical contribution: it clarifies when incomplete data suffice for causal modeling.
- The observed-data objective and ancestral integration provide a training method without a separate imputation stage. Density and counterfactual results on nonlinear models support its practical value.
Critical weaknesses
- The imputation comparison is incomplete. MICE is included, but only one completed dataset is used for downstream training. A comparison with several completions, fitting a CNF to each and combining predictions, is missing. Does MissCNF still have a clear advantage under that setup? Reporting the extra cost would also help judge the value of direct likelihood training.
- The larger-graph experiment reports time and memory usage for up to 64 variables, but not recovery accuracy. The Monte Carlo sample-count study is also limited to small graphs. The results therefore leave open how much sampling is needed as the number of missing ancestors grows. Larger-graph experiments should report density and causal-query accuracy together with runtime.
- The gains are smaller for ATE than for density fitting and counterfactual prediction. The paper reports this difference but does not explain it in much depth. Since causal inference motivates the method, the authors should discuss which queries benefit from the better density estimate and where the improvement remains limited.
Questions for authors
- Does the advantage over MICE persist when several imputations are retained, a CNF is fitted to each, and the predictions are combined? Please include the extra training cost in the comparison.
- For the larger graphs used in the runtime study, how do KL, ATE, and counterfactual errors change as the Monte Carlo budget increases? At what computational cost does accuracy stabilize?
- What explains the smaller gains in ATE estimation? Are there settings where a clear KL improvement brings little or no ATE improvement, and what distinguishes them from settings with larger causal gains?
4 comments
Weakness 2 and Question 2: Accuracy on larger graphs and Monte Carlo budget.
We ran MissCNF on Chain-8, Chain-16, and Erdős–Rényi graphs ER(16,16) and ER(16,64) following DECI [1], with a training size of 1000, MAR with a missing rate of 60%, and from 32 up to 16384 Monte Carlo samples. We report results on Chain-8 in Table 21 on the larger graphs in Table 2, and the imputation baselines for reference in Table 3.
On Chain-8 (Table 1), accuracy improves with . At , MissCNF matches the complete-data model in KL (1.61 vs. 1.62). Training time is almost constant up to , and grows roughly linearly after that. On the larger graphs, we could only run due to GPU memory limits. In this range, the changes are small, as they are on Chain-8, except for a small increase in KL on ER(16,64). We therefore expect larger graphs, which have more missing ancestors per row, to also need in the thousands. We leave this for future work. Even with a moderate budget (Table 3), MissCNF outperforms imputation baselines on Chain-8, Chain-16 and ER(16,16). On ER(16,64), it has the lowest RMSE-ATE and is on par with the best baseline in KL and RMSE-CF.
Table 1. Effect of the Monte Carlo budget on Chain-8 (2 seeds).
| KL | RMSE-CF | RMSE-ATE | Train time | |
|---|---|---|---|---|
| 32 | 3.55 ± 0.30 | 2.46 ± 0.05 | 0.72 ± 0.07 | 0.32h |
| 64 | 3.74 ± 0.24 | 2.46 ± 0.10 | 0.73 ± 0.03 | 0.32h |
| 128 | 3.48 ± 0.66 | 2.44 ± 0.04 | 0.68 ± 0.06 | 0.32h |
| 256 | 3.18 ± 0.82 | 2.40 ± 0.04 | 0.64 ± 0.01 | 0.33h |
| 512 | 3.38 ± 0.37 | 2.39 ± 0.13 | 0.67 ± 0.05 | 0.34h |
| 1024 | 2.88 ± 0.18 | 2.33 ± 0.05 | 0.60 ± 0.05 | 0.35h |
| 2048 | 2.58 ± 0.68 | 2.29 ± 0.01 | 0.56 ± 0.12 | 0.35h |
| 4096 | 1.77 ± 0.20 | 2.20 ± 0.11 | 0.36 ± 0.06 | 1.17h |
| 8192 | 2.09 ± 0.49 | 2.16 ± 0.03 | 0.34 ± 0.06 | 1.78h |
| 16384 | 1.61 ± 0.87 | 2.09 ± 0.06 | 0.29 ± 0.02 | 2.66h |
Table 2. Effect of the Monte Carlo budget on larger graphs.
| Graph | KL | RMSE-CF | RMSE-ATE | Train time | |
|---|---|---|---|---|---|
| Chain-16 (2 seeds) | 32 | 12.69 ± 2.33 | 3.81 ± 0.03 | 1.17 ± 0.00 | 8.55h |
| 64 | 12.34 ± 0.71 | 3.80 ± 0.01 | 1.14 ± 0.04 | 8.49h | |
| 128 | 12.35 ± 2.51 | 3.81 ± 0.03 | 1.17 ± 0.02 | 8.55h | |
| 256 | 12.13 ± 0.34 | 3.80 ± 0.01 | 1.11 ± 0.10 | 8.79h | |
| 512 | 12.23 ± 0.63 | 3.80 ± 0.01 | 1.14 ± 0.05 | 8.59h | |
| ER(16,16) (3 seeds) | 32 | 8.97 ± 0.30 | 2.16 ± 0.09 | 0.72 ± 0.13 | 1.94h |
| 64 | 9.55 ± 1.10 | 2.20 ± 0.01 | 0.72 ± 0.08 | 1.95h | |
| 128 | 8.39 ± 0.20 | 2.13 ± 0.02 | 0.70 ± 0.10 | 1.94h | |
| 256 | 9.46 ± 1.21 | 2.19 ± 0.06 | 0.66 ± 0.18 | 1.98h | |
| 512 | 8.57 ± 0.08 | 2.14 ± 0.04 | 0.71 ± 0.11 | 1.98h | |
| ER(16,64) (3 seeds) | 32 | 18.26 ± 1.04 | 3.74 ± 0.05 | 1.65 ± 0.19 | 6.78h |
| 64 | 18.61 ± 0.67 | 3.78 ± 0.04 | 1.59 ± 0.12 | 6.82h | |
| 128 | 19.72 ± 1.07 | 3.82 ± 0.04 | 1.63 ± 0.22 | 6.87h | |
| 256 | 20.19 ± 0.74 | 3.81 ± 0.05 | 1.66 ± 0.12 | 7.07h | |
| 512 | 20.56 ± 0.66 | 3.84 ± 0.02 | 1.68 ± 0.23 | 6.82h |
Table 3. Comparison with imputation baselines on larger graphs (MissCNF with ). Best non-oracle result in bold.
| Graph | Method | KL | RMSE-CF | RMSE-ATE |
|---|---|---|---|---|
| Chain-8 | Complete data (oracle) | 1.62 ± 0.11 | 2.19 ± 0.05 | 0.61 ± 0.27 |
| MissCNF | 3.18 ± 0.82 | 2.40 ± 0.04 | 0.64 ± 0.01 | |
| MIRACLE | 7.35 ± 0.14 | 2.84 ± 0.01 | 1.12 ± 0.04 | |
| KPI | 8.82 ± 0.19 | 3.03 ± 0.02 | 1.40 ± 0.04 | |
| MICE | 11.70 ± 0.26 | 3.09 ± 0.00 | 1.52 ± 0.03 | |
| Chain-16 | Complete data (oracle) | 12.30 ± 0.07 | 3.79 ± 0.01 | 1.15 ± 0.10 |
| MissCNF | 12.13 ± 0.34 | 3.80 ± 0.01 | 1.11 ± 0.10 | |
| MIRACLE | 16.32 ± 1.15 | 3.94 ± 0.03 | 1.33 ± 0.08 | |
| KPI | 17.78 ± 0.94 | 4.00 ± 0.06 | 1.47 ± 0.07 | |
| MICE | 20.53 ± 1.07 | 3.99 ± 0.03 | 1.31 ± 0.00 | |
| ER(16,16) | Complete data (oracle) | 11.56 ± 4.96 | 2.00 ± 0.06 | 0.62 ± 0.05 |
| MissCNF | 9.46 ± 1.21 | 2.19 ± 0.06 | 0.66 ± 0.18 | |
| MIRACLE | 13.79 ± 1.19 | 2.61 ± 0.10 | 1.20 ± 0.11 | |
| KPI | 13.40 ± 0.60 | 2.77 ± 0.07 | 1.38 ± 0.11 | |
| MICE | 18.76 ± 1.08 | 2.98 ± 0.06 | 1.55 ± 0.05 | |
| ER(16,64) | Complete data (oracle) | 12.16 ± 0.27 | 3.46 ± 0.07 | 1.42 ± 0.08 |
| MissCNF | 20.19 ± 0.74 | 3.81 ± 0.05 | 1.66 ± 0.12 | |
| MIRACLE | 23.29 ± 0.39 | 3.80 ± 0.03 | 1.88 ± 0.03 | |
| KPI | 20.38 ± 0.19 | 3.82 ± 0.02 | 2.07 ± 0.01 | |
| MICE | 24.70 ± 0.12 | 3.83 ± 0.02 | 2.21 ± 0.04 |
[1] T. Geffner, J. Antorán, A. Foster, W. Gong, C. Ma, E. Kıcıman, A. Sharma, A. Lamba, M. Allamanis, N. Pawlowski, et al. (2024). Deep end-to-end causal inference. Transactions on Machine Learning Research (TMLR).
Dear reviewer VnhA,
Since the discussion deadline is approaching, could you kindly confirm if our response has addressed your concerns? If not, we would be happy to answer any follow-up questions and concerns.
Thank you again for your constructive review!
Thank you for your responses. I will keep my original score.
Response to Reviewer VnhA
We thank the reviewer for the positive assessment and the constructive suggestions. Following them, we added a comparison with multiple imputation. The experiments on larger graphs and the Monte Carlo budget are still running, and we will post the results as soon as they are ready. Our answers to each point are below.
Weakness 1 and Question 1: Multiple imputation comparison, with cost.
Thank you for this suggestion. We agree this comparison matters. Appendix A.4 explains why our main experiments use m=1. The downstream object is a full generative model, not a finite-dimensional estimand. As far as we know, no standard Rubin-style rule exists for combining m learned densities. A natural alternative is the equally weighted mixture p~(x)=m1∑jpθ(j)(x).
We use this mixture for KL: to generate n samples, we draw n/m samples from each of the m models. For RMSE-ATE and RMSE-CF: we average the m models' ATE estimates and counterfactual predictions, respectively.
We ran this ensemble (m∈{1,5,10} on Triangle-LIN and Triangle-NLIN, at MAR missingness of rates 30% and 60%, 5 seeds. Table 1 reports three metrics and total wall-clock cost (imputation plus all m CNF fits).
Table 1. MissCNF vs. an m-completion MICE ensemble. Mean ± std over 5 seeds; cost is imputation + all m CNF-fit wall-clock time, in seconds. Best KL, RMSE-CF and RMSE-ATE per setting in bold.
The results match Experiment 1, where MICE does best on linear tasks and worst on nonlinear ones. On Triangle-LIN, where the MICE model is correctly specified, more imputations lower the ensemble's KL (0.007→0.002→0.001 at p=0.3). On Triangle-NLIN, increasing m from 1 to 10 brings no improvement. Averaging over m imputations reduces the variance from any single imputation, but it does not fix the bias that comes from MICE's misspecified conditional models. MissCNF is slower (370–430 s vs. approximately 80 s for m=10), but on nonlinear tasks the extra cost buys a large gain in accuracy. We will add this experiment to Appendix A.4.
Weakness 2 and Question 2: Accuracy on larger graphs and Monte Carlo budget.
We agree this is important, we're currently rerunning these experiments and will report the results as soon as possible.
Weakness 3 and Question 3: Why are ATE gains smaller than density and counterfactual gains?
We thank the reviewer for this question. ATE depends only on the means of two interventional distributions, while KL and counterfactuals depend on the full shape of the distribution. A baseline model can recover the means well while distorting variances, making the benefit of our method stronger in KL and counterfactual error, but much less in ATE.
However, lower KL still corresponds to lower causal error. By Javaloy et al. (2023, Thm. 1), recovering the joint identifies interventional and counterfactual queries, and this holds empirically across all 540 runs in Experiments 1–4. The Spearman correlation between KL and RMSE-ATE is ρ=0.63, and between KL and RMSE-CF it is ρ=0.73 (both p<10−59). KL tracks counterfactual error more closely than ATE error in every experiment, which matches the explanation above. We will add this discussion and the correlation analysis to the paper.