- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper shows that causal foundation models produce poorly calibrated uncertainty intervals for average treatment effects and subgroup effects. The main issue is that the predictions are correlated and the variance estimates considers them as independent. The authors theoretically characterize this problem and propose two neural-process models that account for these correlations.
Critical strengths
-
The paper identifies an important limitation of existing CFMs and demonstrates it across six standard benchmarks.
-
The theoretical analysis clearly explains how correlated estimation errors lead to poor interval coverage.
-
The experiments compare independent and joint aggregation using the same trained models and show that accounting for correlations substantially improves coverage.
Critical weaknesses
-
The proposed models are evaluated only on a simple synthetic setting with three covariates. They are not evaluated on standard benchmarks such as IHDP and ACIC 2016, where the paper demonstrates poor coverage for CausalPFN.
-
The main theoretical result is based on a standard variance decomposition for correlated errors, closely related to the well-known design effect in survey sampling. While applying this analysis to causal foundation models is useful, the paper could better clarify what is theoretically novel beyond this application.
Questions for authors
Can the authors evaluate their proposed methods on standard benchmarks such as IHDP and ACIC 2016? Since the proposed models are tested only on a relatively simple synthetic setting, it is unclear whether the improvements in coverage generalize to more realistic problems.
Response to Reviewer Prsf
Thank you for the review.
W1 and Q: the models improve coverage on IHDP and ACIC 2016. As heads on a frozen CausalPFN backbone (general response), their full-sample coverage on IHDP is 0.94 for the CNP head and 0.89 for the latent NP head, against 0.60 for CausalPFN. On ACIC 2016, coverage is 0.71 and 0.78, against 0.20 for CausalPFN. Unit-level aggregation from the same CNP and latent NP head weights gives 0.30 and 0.48 on IHDP and 0.10 and 0.18 on ACIC 2016, so the gain comes from the aggregation step (Figure 3 in the revision).
W2: the new theory is the source of the error correlation and its consequence. Proposition 1 is the design effect, and the paper credits it to Kish (1965). The new theoretical content is Proposition 2. The error correlation equals the posterior covariance of unit effects over the SCM, so every CFM that reports unit-level distributions has it. The unit-level marginals do not determine this covariance, so no interval built from them alone can account for it. The new exact-posterior experiment shows the consequence directly. Unit-level aggregation of the exact posterior's marginals has coverage of 0.25 at k = n on our synthetic family, while the exact posterior of the subset-CATE has nominal coverage (Figure 2 in the revision). The revised paper states these points more directly in Section 4.