- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes CausalIDView, a causal effect estimation from observational data benchmark to evaluate causal estimators on different causal identification regimes (back-door, front-door, instrumental variable, proximal). It is motivated by recent developments on causal foundation models (CFMs) and their differences in pre-training that depend on different identification conditions. CausalIDView’s different identification regimes address this gap, shedding light on shortcomings of current CFMs and on their differences as compared to standard estimation approaches.
Critical strengths
Causal effect estimation from different identification conditions is an important problem. This is especially important in the context of CFMs as their pre-training data may use different identification conditions, which can ultimately bias their predictions at deployment. Thus, comparing CFMs fairly across different identification regimes is a good step towards a better understanding of them and their limitations.
Strong experimental design. On top of multi-regime data, the paper also covers additional semi-synthetic and RCT datasets. It also covers causal estimators very comprehensively, including established and state-of-the-art estimation methods, as well as the modular approaches that combine TabPFN with identification specific techniques.
Critical weaknesses
Mostly presentation issues (all addressable). The paper relies too much on the appendix in some places in the main content of the paper. My understanding is that information in the appendix is optional for the reader and should be treated as such. Some examples. RQ3 in section 6 references Figure 6 that is available only in the appendix (it is the main evidence source for this paragraph). Tables 1 and 2 report results for methods never mentioned in the main text (e.g. Causal Tree, TARNet, GRF, CFRNet). I know these are described in the appendix, but some description should also appear in the main text.
Questions for authors
- I would suggest to either move RQ3 from section 6 to the appendix, or move Figure 6 to the main body of the paper.
- I would suggest to add missing descirptions of the methods included in Tables 1 and 2 in the main body of the paper.
3 comments
Thanks for the revision
Thank you for addressing my concerns. I have also read the issues raised by the other reviewers, all the authors' responses and the updated manuscript. In my view, the new additions and results, as summarised in the global response, make this paper much stronger. I am overall impressed by the revision and will update my score to 4 to reflect the now much stronger state of the paper.
Thank you very much for your thoughtful reassessment and for raising your score. We are delighted that you find the revised manuscript much stronger. Your insightful comments played an important role in shaping these improvements, and we sincerely appreciate your time and engagement throughout the review process.
Clarifications to Baseline Descriptions and RQ3 Findings
We sincerely thank the reviewer for the careful assessment of our manuscript and the constructive suggestions. We agree that the original manuscript did not sufficiently present the supporting evidence for RQ3 and the descriptions of certain baselines in the main text.
To address these concerns, we have revised Section 5.2 to include brief descriptions of the previously omitted baselines and refined the RQ3 discussion in Section 6 to more clearly present its supporting evidence and findings in the main text. These revisions are highlighted in blue in the revised PDF for ease of reference. We appreciate the reviewer's valuable feedback, which has helped us improve the clarity and readability of our manuscript.