- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper looks at how to improve tabular foundation models for treatment effect estimation. The authors argue that these models are still hurt by the imbalance between treated and control groups. They keep the foundation model frozen and add a trainable encoder in front of it, trained with a Wasserstein balancing loss. A small task-specific predictor is added on top of the encoder, so that balancing does not remove outcome information. A learnable gate then combines the outputs of this predictor and the foundation model. The authors also give an error bound for the method. Experiments on synthetic data, IHDP, and ACIC 2016 with four different backbones show lower estimation error in most cases.
Critical strengths
On originality, the question itself is timely. Causal foundation models are new, and how to adapt them to one dataset without fine-tuning has not been studied much. Using an external encoder plus a light side predictor around a frozen in-context learner is a reasonable and fairly creative combination of representation balancing (in the style of CFR) and model ensembling. The over-balancing argument also gives a clear reason for the side predictor. The authors do not just add it as an extra module.
Critical weaknesses
My main concern is the strength of the standalone baselines. Several numbers are far worse than what is usually reported. CausalFM gets on IHDP and above 2.0 on the simulation. Do-PFN gets about 6.6 on IHDP. DESCN and CausalOT get above 7 on IHDP, which is worse than KNN. These models were published with much better results on similar benchmarks. This makes me suspect a setup problem, such as outcome scaling, how treatment is passed as input, or using Do-PFN for a target it was not built for. If the backbones are broken, then most of the gain of tecFM-CausalFM (8.6 to 0.53) probably comes from the task-specific predictor and not from “enhancing” the foundation model. The paper should explain exactly how each baseline was run and why its numbers differ from the original papers.
Questions for authors
The theory has several gaps. In the proof of Theorem 1, the four counterfactual terms appear. Only one is discussed, with the phrase “it is easy to prove ”. As written this is not correct. Lemma 3 would give a counterfactual error bounded by the factual error plus a Lipschitz constant times , not alone. The step also needs the loss to be Lipschitz in representation space, and usually needs to be invertible, as in Shalit et al. (2017). Neither is assumed. There is also a mismatch: the lemma is stated on representation distributions, but the errors are integrated over .
3 comments
Question 1. Correcting the counterfactual-error bound and specifying its assumptions.
Answer: We thank the reviewer for carefully examining the proof. We address the four questions below and provide the corrected proof afterward.
-
Control of the four counterfactual terms. We apply Lemma 3 separately to the two weighted losses in Eqs. (16)--(17), keeping the predictor and treatment condition fixed and changing only the group over which the loss is evaluated. Thus, and are compared with their factual errors in the control group, while and are compared with their factual errors in the treated group. The four inequalities are given explicitly in the proof below.
-
Correction of the counterfactual-error step. Lemma 3 bounds the difference between the two group-specific errors, not the counterfactual error alone. The corrected step is therefore , where and are the representation distributions defined in Eq. (8).
-
Regularity and invertibility. For fixed predictors, gate, and context, we assume that each weighted loss in Eqs. (16)--(17) can be written as and satisfies the condition of Lemma 3: . This does not require to be invertible. It requires that inputs with the same representation have the same value of each weighted loss. We will make these assumptions explicit in Theorem 1.
-
Covariate and representation distributions. We use for the covariate distribution in Eqs. (16)--(17) and for its representation distribution, as in Eq. (8). For each weighted loss, the two integrals are equal: . This identity allows Lemma 3 to be applied in representation space to the errors defined over the original covariates.
Proof of Theorem 1.
Let denote the covariate distribution conditional on , and let denote the corresponding representation distribution defined in Eq. (8). Fix the predictors, gate, and context. For each , consider the two weighted losses in Eqs. (16)--(17):
By assumption, each loss can be written as for an integrable function satisfying the condition of Lemma 3, . For either choice of and each ,
Hence, Lemma 3 bounds the absolute difference between the treated and control expectations of the same weighted loss by . Applying this bound to each of the two losses under each treatment condition gives
We now apply these inequalities to the PEHE. Using , the gated squared-error identity in Eq. (28), and the population mixture , we obtain
By Eq. (29), . Dropping these nonnegative terms from the subtraction and substituting the four inequalities above gives two transport terms in each group. Since , their total contribution is . Therefore,
which establishes the population bound in Eq. (18) under the stated conditions.
Dear Reviewer 2ux9,
Thank you again for your time and valuable feedback. We would appreciate any further comments on our responses and revised manuscript. If you feel that our responses have addressed your concerns, we would be very grateful for your support for our paper during the discussions with the Area Chair.
Best regards,
The Authors
Weakness 1. Reliability of standalone baselines and whether gains mainly come from the task-specific predictor.
Answer: We thank the reviewer for raising this concern. Studies [1,2] also report relatively large standalone errors for Do-PFN and CausalFM on IHDP and ACIC 2016. These studies use the same benchmarks and evaluation metric, but their evaluation pipelines are not identical to ours. We use the same predefined IHDP train/test data as the public CausalPFN benchmark. On ACIC, however, their public loader uses integer encoding for categorical covariates, whereas ours uses one-hot encoding. The published results therefore provide context rather than a comparison under identical settings. The contribution of the task-specific predictor is examined separately through the branch ablations below.
To examine whether the gain comes solely from the task-specific predictor, we report the mean learned gate weights below. A smaller gt assigns more weight to the foundation model, while a larger gt assigns more weight to the task-specific predictor. The results show that the foundation model retains a substantial weight across these settings. In summary, the predictor of our task-specific model usually enhances the performance of the base model.
[1] Balazadeh et al. CausalPFN: Amortized Causal Effect Estimation via In-Context Learning. 2025. Appendix D.8, Table 8.
[2] Reuter et al. Use What You Know: Causal Foundation Models with Partial Graphs. 2026. Preprint, arXiv:2602.14972. Appendix J.2, Table 3.