- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a causal foundation model based on mixture of experts architecture that takes different mechanism representations from a tabpfn model and learns a router on top of them. Empirical evaluation is conducted against other causal models, both amortized and non-amortized baselines and across various benchmarks.
Critical strengths
For wider adoption of causal foundation models I believe it is important we have better interpretability of their inference - this paper recognizes this need and takes first steps in that direction. I also appreciate that it reuses representations from an already trained tabpfn, which is potentially a more scalable approach than training a causal TFM from scratch.
Critical weaknesses
I have several concerns: Contribution: Contribution section is unnecessarily long, making it paradoxically less clear what is the novelty. To my understanding the motivations include 1) method-specific assumptions while at the same acknowledging that is not a concern for end-to-end inference amortizing foundation models which leave choice of method implicit and up to the model, 2) causal identification and inference separation which is already followed by CausalPFN and CausalFM and is not novel (admittedly DoPFN does not do this), or - generally - the motivation just conflates statistical modelling choices with assumptions required for causal identification, 3) fundamental problem of causal inference making counterfactual information unavailable while this is not actually true for synthetic data training of pfns (this is exactly why causal pfns exist and work so well), and 4) confusingly claims not to be yet another CFM after primarily positioning the work as so.
Methodology: 1) The paper does not convincingly explain why correlations between embedding distances and factual outcome differences should identify representations suitable for causal effect estimation - especially if contribution is built on TTA then simple correlation might not suffice (Representations preserving factual outcome similarity need not preserve the confounding or effect-modifying information required for accurate causal effect estimation). 2) MoE-Learner underperforms its own TabPFN backbone on IHDP and ACIC despite introducing substantial additional machinery which is a concern amplified by the fact that 3) the foundation-model comparisons lack sufficient information about matching pretrained checkpoints, inference configurations, and computational budgets so it is unclear if comparisons are fair. Finally I am left unconvinced by the ablations.
In sum, the contribution is in my view less substantial than presented, theoretical motivation is partially lacking and the empirical evidence does not convincingly establish the benefit of adapting tabpfn representation as such over simply using default tabpfn.
Questions for authors
For my major points please see the weaknesses section about. Additionally, could the authors please provide an anonymized code repository for inspection?
3 comments
Question 4
- Ablation and code.
We ensure a fair comparison with other foundation models. For CausalPFN, we directly used their Table 1; moreover, we also reproduced their whole experimental pipeline using their official code in https://github.com/vdblm/CausalPFN/ and no significant difference was observed. For CausalFN and Do-PFN, their official codebases were adopted:
https://github.com/yccm/CausalFM
https://github.com/jr2021/Do-PFN
The exact datasets and conda environments were used to run the two methods. We first reproduced their results in their papers to ensure the environments were correctly configured. Then we exported the exact data from CausalPFN and imported them into CausalFM/Do-PFN to produce the results. Importantly, since we also had doubts about whether we had run these foundation models correctly, we investigated recently published papers where these foundation models were used/tested. Please see page 40 of reference [2] and page 33 of reference [3]: Their reported numbers from CausalFM and/or Do-PFN are very close to the numbers we report in our paper. The code to reproduce our results is uploaded in a zip file.
[1] Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation
[2] CausalPFN: Amortized Causal Effect Estimation via In-Context Learning
[3] Use What You Know: Causal Foundation Models with Partial Graphs
Having considered the authors' rebuttal and the concerns raised by the other reviewers, I am lowering my score to 1 (Clear rejection). Unfortunately, the rebuttal has not resolved the fundamental issues with this submission and, in several respects, has reinforced my concerns. The paper suffers from an insufficiently motivated and poorly differentiated contribution, serious conceptual confusion regarding causal identification and estimation, a lack of theoretical justification for its central methodological innovation, and empirical evidence that fails to convincingly demonstrate its value over substantially simpler alternatives. The additional experiments do not meaningfully address these shortcomings, while the authors explicitly acknowledge that their theoretical analysis does not justify the proposed routing criterion. Taken together, I no longer consider these merely weaknesses in an otherwise promising contribution, but fundamental shortcomings in the scientific soundness and significance of the work. I therefore do not believe the paper meets the standards for acceptance at ICLR.
Thank you for your comments!
Thanks for your thoughtful review. We fully agree with you that interpretability of causal inference is important, and our work is only an initial step. But we hope our work can shed light on future research. We address your concerns as follows.
Contribution.
Your summarization of our contributions is basically accurate. Specifically, our work addresses a question largely neglected by existing causal estimation methods: when counterfactual labels are unavailable, how can we understand whether, why, and when a causal estimation strategy is suitable for the observed data? To our best knowledge, no previous work has tackled this important problem. One earlier related work is [1]; however, as its title suggests, it is only an empirical analysis and still didn’t answer “why”. In this work, instead, we provide mechanistic interpretability - it examines the data geometry through representations and uses the alignment between representation distance and outcome difference to provide an interpretable, data-dependent basis for selection. We will reorganize the contribution section as numbered items following your summarization in the final version.
Methodology and theorems.
You are absolutely right to ask this question: “Why correlations between embedding distances and factual outcome differences should identify representations suitable for causal effect estimation?” Theorem 1 shows that (under the stated local smoothness conditions), nearby representations can support meaningful KNN-style outcome estimation. However, we are unable to (and didn’t) mathematically prove any theorem for the use of the Spearman correlation. Theorem 1 does not provide useful signal for the selection; and we clearly stated that the correlation is just a useful “empirical trend” (page 7). We also tested other metrics in Table 5. In conclusion, we proved what we were able to prove (Theorem 1), and left what we are unable to prove to empirical. But we will continue working on the theoretical part to try to fully answer your question above in future work.
Performance.
As shown in the supplementary Figures 5 and 6, lower performance than the TabPFN backbone is due to higher variance, but mostly comparable. However, does our method truly outperform a plain TabPFN? We had the same doubt as you. Therefore, we conducted two new experiments on front-door adjustment and Instrumental Variables (IV). First, we cloned the exact same datasets, codebase, and conda environment of CausalFM [2] and reproduced its scores. We also ran Do-PFN, CausalPFN, TabPFN and compare them with CausalFM. Finally, we ran our MoE-Learner using the same setting. The results are shown as follows.
Front-door
Binary IV
Please note that while PFNs (CausalPFN and TabPFN) are not specifically designed for front-door and IV, they show competitive performance, especially TabPFN; although it is a general-purpose tabular foundation model, it outperforms all causal foundation models. However, our method, MoE-Learner, is even better than TabPFN on both front-door and IV. These new experimental results significantly strengthen our method, making it not only the first unified MoE framework for causal effect estimation but also a strong empirically validated estimator even with unobserved confounding.