- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper considers the problem of causal sensitivity under different levels of hidden confounding, rather than point estimation. Traditional methods for causal sensitivity require large amounts of computation for any changes in the dataset, causal query, or level of confoundedness of interest. Hence, this work builds on prior work that trains foundation models using prior data-fitted networks for causal effect estimation, and considers this problem for causal sensitivity. However, in causal sensitivity, labels are not immediate given a structural causal model or data generating process, so the paper considers a new approach to getting upper and lower bound labels given a ground-truth DGP and causal query, and who theoretical strengths of their method. Empirically, the paper shows that their approach performs well across different held-out DGPs, causal queries, and sensitivity levels. Overall, the paper claims this is the first foundation model for in-context automatic causal sensitivity analysis.
Critical strengths
-
This is a new and extremely well-motivated problem. There has been more interest in recent work about how prior data-fitted networks could work for causal effect estimation, but no one has tackled the harder problem of understanding how to do this for causal sensitivity analysis. The paper lays out the motivation for this work well and describes an extremely sensible approach to achieving this.
-
The label engineering choice is neat, and the choice of warm starting was smart and was proven correct both through theory and empirical results. I appreciate that the theory was able to build a realistic mechanism for quick label definitions that were used for training.
Critical weaknesses
- The primary weakness is the limited evaluation of their methods. Primarily, I have three issues with the evaluation:
- The label construction mechanism is never validated. The paper should consider whether they can evaluate how good their label construction technique is to the closed form solution with MSM.
- For MSM, there should be stronger baselines that directly estimate the nuissance functions required for the closed-form solution. One option would be TabPFN, which the paper notes as related work, which could be used to obtain quantiles for the conditional outcome distribution. In general, the baselines are quite weak with respect to what else could be done in the simpler MSM case.
- For the KL sensitivity model, the procedure for generating labels during training mirrors that for the labels used for evaluation (though evaluation has a higher budget). Though no data is shared, it seems quite circular to assert the model is better given the labels were estimated the same way for both. I am not super familiar with the field, but are there alternative, potentially high-compute methods that are not their proposed method to create labels for the held-out eval set for this model, so that we can ensure the high performance is not simply the alignment in how labels are created?
- The model is only trained for a limited set of covariates (10), and only evaluated on synthetic datasets that the paper has generated. The paper should consider using semi-synthetic data like IHDP or ACIC, and introducing hidden confounding in some way so that causal sensitivity is needed. This would also stress test the method that was built for a limited context size and limited set of covariates to a somewhat more realistic set of covariates.
Questions for authors
- What is the potential for better performance to be a result of a shared label-construction approach during training and eval? Are there other ways to construct labels separate from how they were build for training that can be used for the eval?
- How well do the learned labels from the proposed method compare to the MSM closed-form solution labels?
- How does your approach fare against baselines that estimate nuisance parameters of the closed-form solution for the MSM model?
- How do the learned models scale to new, unseen, realistic sets of covariates and datasets beyond the DGPs created synthetically?
5 comments
Thank you for the new experiments and answering the questions I posed in the review. My answers have mainly been answered, but some of the answers bring up more concerns rather than clarification. I believe my questions around the KL validation and analytical MSM bounds have been answered.
However, the new results on the realistic covariates are quite concerning to me. First, the proposed TabPFN method performs substantially better on the realistic ACIC covariates compared to the proposed method, so NeuralCSA is not the right baseline. I agree that it seems the benefit to the proposed FM is for in-prior datasets. However, this limits the scope, novelty, and applicability of the approach significantly with respect to what the paper claims it can do. Specifically, applying this to unseen or held-out data distributions shows a large degradation compared to baselines, which limits the applicability of the model substantially. This is especially true given the fact that the DGP behind real covariates is unknown, so when applying to real data, we won't know whether the new dataset will be in-prior. Combined with the fact that it does not transfer well to datasets with more covariates, I worry that the scope of the method has been greatly diminished.
Hence, I will still keep my original score.
Dear reviewer,
Thank you for your response. We would like to respectfully point out that the TabPFN baseline is only applicable for the MSM-model case, where an analytical form of the estimand is available. This is in contrast to the general case where the estimand is an implicit solution to an optimization problem (and so tabpfn baseline is not possible; see Table 10, page 34) - in the general case our method remains the only amortized solution that successfully leverages deep learning scaling.
More broadly, we would like to argue that the value of our contribution goes beyond chasing sota on every metric. We are the first to propose a foundation model for general causal sensitivity problems, showing competitive performance overall, and where not currently sota, our model is best positioned to benefit from further naive scaling of the underlying transformer (please note we do not possess the computational budget to compete with prior labs and do not believe that is the purpose of our work in the first place). We will adjust the framing of the paper to more clearly outline our contribution.
We would like to kindly ask the reviewer to reconsider raising their score. We are happy to make any final changes necessary to assist this.
Thank you for recognizing the motivation and label-construction contribution, and for your detailed evaluation suggestions. These motivated additional checks against independent references and comparisons with stronger baselines. Changes to PDF are marked in blue.
(RMSE pairs below report lower/upper endpoint errors in units of observed-outcome SDs.)
W1/Q2. Label construction versus analytical MSM bounds.
We now validate the general Lagrangian label optimizer in MSM mode against analytical bounds on 16 fresh DGPs, comparing labels and analytical bounds at the same sensitivity level. The existing MSM PFN uses analytical training labels.
In a matched four-DGP comparison, increased computation reduces label RMSE from 0.034/0.044 to 0.016/0.019. These results directly validate numerical label accuracy against known sharp bounds, with additional computation more than halving the remaining inward error.
W2/Q3. Stronger MSM nuisance-estimation baselines.
We now implement TabPFN nuisance estimation followed by analytical MSM bounds. TabPFN estimates propensity and outcome distributions through in-context inference; PI-FM predicts bounds directly. We also compare B-Learner and Dorn–Guo on their respective CATE and ATE targets.
On matched in-prior CAPO predictions, PI-FM's RMSE is 0.296/0.296, versus TabPFN's 0.406/0.391. On ACIC-d10, TabPFN is stronger: 0.105/0.106, versus 0.218/0.223. The stronger baseline therefore confirms our model's in-prior accuracy advantage and identifies where nuisance estimation offers better transfer.
W3/Q1. Independent KL validation and shared-solver error.
We now add two references independent of the training solver. Exact Gaussian-family bounds give analytical ground truth: optimizer-label RMSE is 0.03–0.08 for Γ≤2.5.
Separately, on 64 fresh Gaussian-conditional DGPs, the frozen PFN's outward interval contains the entire exact reference interval 90.2% of the time against a nominal 90% joint-containment target. Individual endpoint coverage remains below nominal 90% at 82.7%/83.6%. This supports outward reliability independently of shared-solver agreement.
For IHDP, we compute reference bounds using an independent convex solver. Refining the numerical approximation changes checked endpoints by less than 0.001 observed-outcome SDs. These complementary checks substantially strengthen the KL evaluation by validating predictions against independent references in both analytical and real-covariate settings.
W4/Q4. Transfer to realistic covariates and higher dimensions.
We now test frozen models on real ACIC-d10, ACIC-d82 and IHDP-d25 covariates, simulating treatment and outcomes with hidden confounding to retain computable reference bounds.
MSM's nominal 90% endpoint coverage is 92.4%/90.4%, 70.7%/68.0% and 82.8%/81.8%, respectively. ACIC-d10 demonstrates near-nominal calibration on real covariates; the full-feature evaluations identify calibration limits. PI-FM also improves endpoint accuracy over fitted NeuralCSA on every matched MSM/KL cohort. Together, these results demonstrate useful transfer beyond synthetic training covariates.