acceptodds
Under review as a conference paper at ICLR 2027

Enhancing causal effect foundation model with task-specific model

Abstract

Causal effect estimation aims to evaluate the effects of a treatment and plays a vital role in decision-making. Benefiting from in-context learning, tabular foundation models have shown promising performance for causal effect estimation without fine-tuning on task-specific context data. Despite their strong generalization ability to new tasks, foundation models cannot fully address the distribution shift between treatment groups caused by confounding bias, which limits the performance of causal effect estimation. One natural approach is to learn an encoder to explicitly align the representations of groups. However, enforcing representation alignment could lead to over-balancing, which means that predictive information for outcome prediction is partially removed. Although minimizing the prediction loss of the foundation model can provide guidance for learning the encoder to some extent, the complex architecture of the foundation model makes it difficult to effectively propagate task-relevant gradient signals from the output back to the encoder. To address this issue, we incorporate a lightweight task-specific predictor after the encoder to preserve outcome-relevant predictive information in the learned representations. By doing this, the foundation model is enhanced by the task-specific encoder and predictor trained on context data. Moreover, we design a gated aggregation mechanism to adaptively fuse the predictions of the foundation model and the task-specific model. We theoretically analyze the estimation error of our method, and conduct extensive experiments based on multiple backbones of foundation models. The experimental results on benchmark datasets demonstrate the effectiveness of our method and its general applicability across different backbones.

Primary area:
causal reasoning
Keywords:
Causal Effect Estimation, Tabular Foundation Models, In-Context Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2ux9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how to improve tabular foundation models for treatment effect estimation. The authors argue that these models are still hurt by the imbalance between treated and control groups. They keep the foundation model frozen and add a trainable encoder in front of it, trained with a Wasserstein balancing loss. A small task-specific predictor is added on top of the encoder, so that balancing does not remove outcome information. A learnable gate then combines the outputs of this predictor and the foundation model. The authors also give an error bound for the method. Experiments on synthetic data, IHDP, and ACIC 2016 with four different backbones show lower estimation error in most cases.

Critical strengths

On originality, the question itself is timely. Causal foundation models are new, and how to adapt them to one dataset without fine-tuning has not been studied much. Using an external encoder plus a light side predictor around a frozen in-context learner is a reasonable and fairly creative combination of representation balancing (in the style of CFR) and model ensembling. The over-balancing argument also gives a clear reason for the side predictor. The authors do not just add it as an extra module.

Critical weaknesses

My main concern is the strength of the standalone baselines. Several numbers are far worse than what is usually reported. CausalFM gets ϵPEHE≈8.6\sqrt{\epsilon_{\text{PEHE}}}\approx 8.6 on IHDP and above 2.0 on the simulation. Do-PFN gets about 6.6 on IHDP. DESCN and CausalOT get above 7 on IHDP, which is worse than KNN. These models were published with much better results on similar benchmarks. This makes me suspect a setup problem, such as outcome scaling, how treatment is passed as input, or using Do-PFN for a target it was not built for. If the backbones are broken, then most of the gain of tecFM-CausalFM (8.6 to 0.53) probably comes from the task-specific predictor and not from “enhancing” the foundation model. The paper should explain exactly how each baseline was run and why its numbers differ from the original papers.

Questions for authors

The theory has several gaps. In the proof of Theorem 1, the four counterfactual terms appear. Only one is discussed, with the phrase “it is easy to prove E0,h(1)≤WE^{(1)}_{0,h}\le W”. As written this is not correct. Lemma 3 would give a counterfactual error bounded by the factual error plus a Lipschitz constant times WW, not WW alone. The step also needs the loss to be Lipschitz in representation space, and usually needs ϕ\phi to be invertible, as in Shalit et al. (2017). Neither is assumed. There is also a mismatch: the lemma is stated on representation distributions, but the errors are integrated over XX.

3 comments

Weakness 1. Reliability of standalone baselines and whether gains mainly come from the task-specific predictor.

Answer: We thank the reviewer for raising this concern. Studies [1,2] also report relatively large standalone errors for Do-PFN and CausalFM on IHDP and ACIC 2016. These studies use the same benchmarks and evaluation metric, but their evaluation pipelines are not identical to ours. We use the same predefined IHDP train/test data as the public CausalPFN benchmark. On ACIC, however, their public loader uses integer encoding for categorical covariates, whereas ours uses one-hot encoding. The published results therefore provide context rather than a comparison under identical settings. The contribution of the task-specific predictor is examined separately through the branch ablations below.

SourceModelIHDPACIC 2016
[1], Table 8Do-PFN6.074.11
[2], Table 3Do-PFN6.074.11
OursDo-PFN6.684.54
[2], Table 3CausalFM5.863.30
OursCausalFM8.724.99

To examine whether the gain comes solely from the task-specific predictor, we report the mean learned gate weights below. A smaller gtg_t assigns more weight to the foundation model, while a larger gtg_t assigns more weight to the task-specific predictor. The results show that the foundation model retains a substantial weight across these settings. In summary, the predictor of our task-specific model usually enhances the performance of the base model.

DatasetICL modelgˉ0\bar g_0gˉ1\bar g_1
ACICTabICL0.0850.117
CausalPFN0.0370.043
DoPFN0.1870.216
CausalFM0.6290.703
IHDPTabICL0.2560.533
CausalPFN0.2220.489
DoPFN0.5740.674
CausalFM0.6950.730

[1] Balazadeh et al. CausalPFN: Amortized Causal Effect Estimation via In-Context Learning. 2025. Appendix D.8, Table 8.

[2] Reuter et al. Use What You Know: Causal Foundation Models with Partial Graphs. 2026. Preprint, arXiv:2602.14972. Appendix J.2, Table 3.

Question 1. Correcting the counterfactual-error bound and specifying its assumptions.

Answer: We thank the reviewer for carefully examining the proof. We address the four questions below and provide the corrected proof afterward.

  1. Control of the four counterfactual terms. We apply Lemma 3 separately to the two weighted losses in Eqs. (16)--(17), keeping the predictor and treatment condition fixed and changing only the group over which the loss is evaluated. Thus, E0,h(1)\mathcal E^{(1)}_{0,h} and E0,f(1)\mathcal E^{(1)}_{0,f} are compared with their factual errors in the control group, while E1,h(0)\mathcal E^{(0)}_{1,h} and E1,f(0)\mathcal E^{(0)}_{1,f} are compared with their factual errors in the treated group. The four inequalities are given explicitly in the proof below.

  2. Correction of the counterfactual-error step. Lemma 3 bounds the difference between the two group-specific errors, not the counterfactual error alone. The corrected step is therefore E0,h(1)≤E0,h(0)+W(C,P1ϕ,P0ϕ)\mathcal E^{(1)}_{0,h}\le \mathcal E^{(0)}_{0,h}+W(C,P_1^\phi,P_0^\phi), where P1ϕP_1^\phi and P0ϕP_0^\phi are the representation distributions defined in Eq. (8).

  3. Regularity and invertibility. For fixed predictors, gate, and context, we assume that each weighted loss in Eqs. (16)--(17) can be written as m(ϕ(x))m(\phi(x)) and satisfies the condition of Lemma 3: ∣m(ϕ(x))−m(ϕ(x′))∣≤C(ϕ(x),ϕ(x′))|m(\phi(x))-m(\phi(x'))|\le C(\phi(x),\phi(x')). This does not require ϕ\phi to be invertible. It requires that inputs with the same representation have the same value of each weighted loss. We will make these assumptions explicit in Theorem 1.

  4. Covariate and representation distributions. We use Pt′P_{t'} for the covariate distribution in Eqs. (16)--(17) and Pt′ϕP_{t'}^\phi for its representation distribution, as in Eq. (8). For each weighted loss, the two integrals are equal: ∫Xm(ϕ(x)),dPt′(x)=∫ϕ(X)m,dPt′ϕ\int_{\mathcal X}m(\phi(x))\\,dP_{t'}(x)=\int_{\phi(\mathcal X)}m\\,dP_{t'}^\phi. This identity allows Lemma 3 to be applied in representation space to the errors defined over the original covariates.

Proof of Theorem 1.

Let PtP_t denote the covariate distribution conditional on T=tT=t, and let PtϕP_t^\phi denote the corresponding representation distribution defined in Eq. (8). Fix the predictors, gate, and context. For each t∈0,1t\in\\{0,1\\}, consider the two weighted losses in Eqs. (16)--(17):

gt(x)(ht(x)−μt(x))2,(1−gt(x))(ft∗(x)−μt(x))2.g_t(x)(h_t(x)-\mu_t(x))^2,\qquad (1-g_t(x))(f_t^*(x)-\mu_t(x))^2.

By assumption, each loss can be written as m(ϕ(x))m(\phi(x)) for an integrable function mm satisfying the condition of Lemma 3, ∣m(ϕ(x))−m(ϕ(x′))∣≤C(ϕ(x),ϕ(x′))|m(\phi(x))-m(\phi(x'))|\le C(\phi(x),\phi(x')). For either choice of mm and each t′∈0,1t'\in\\{0,1\\},

∫Xm(ϕ(x)),dPt′(x)=∫ϕ(X)m,dPt′ϕ.\int_{\mathcal X}m(\phi(x))\\,dP_{t'}(x)=\int_{\phi(\mathcal X)}m\\,dP_{t'}^\phi.

Hence, Lemma 3 bounds the absolute difference between the treated and control expectations of the same weighted loss by W(C,P1ϕ,P0ϕ)W(C,P_1^\phi,P_0^\phi). Applying this bound to each of the two losses under each treatment condition gives

E1,h(0)≤E1,h(1)+W(C,P1ϕ,P0ϕ),\mathcal E^{(0)}_{1,h}\le \mathcal E^{(1)}_{1,h}+W(C,P_1^\phi,P_0^\phi),

E1,f(0)≤E1,f(1)+W(C,P1ϕ,P0ϕ),\mathcal E^{(0)}_{1,f}\le \mathcal E^{(1)}_{1,f}+W(C,P_1^\phi,P_0^\phi),

E0,h(1)≤E0,h(0)+W(C,P1ϕ,P0ϕ),\mathcal E^{(1)}_{0,h}\le \mathcal E^{(0)}_{0,h}+W(C,P_1^\phi,P_0^\phi),

E0,f(1)≤E0,f(0)+W(C,P1ϕ,P0ϕ).\mathcal E^{(1)}_{0,f}\le \mathcal E^{(0)}_{0,f}+W(C,P_1^\phi,P_0^\phi).

We now apply these inequalities to the PEHE. Using (a−b)2≤2a2+2b2(a-b)^2\le 2a^2+2b^2, the gated squared-error identity in Eq. (28), and the population mixture p(x)=P(T=1)p(x∣T=1)+P(T=0)p(x∣T=0)p(x)=P(T=1)p(x\mid T=1)+P(T=0)p(x\mid T=0), we obtain

ϵPEHE=∫X[(h~1(x)−μ1(x))−(h~0(x)−μ0(x))]2p(x),dx\epsilon_{\mathrm{PEHE}}=\int_{\mathcal X}\bigl[(\widetilde h_1(x)-\mu_1(x))-(\widetilde h_0(x)-\mu_0(x))\bigr]^2p(x)\\,dx

≤2∑t=01∫X(h~t(x)−μt(x))2p(x),dx\le 2\sum_{t=0}^{1}\int_{\mathcal X}(\widetilde h_t(x)-\mu_t(x))^2p(x)\\,dx

=2∑t=01∑t′=01P(T=t′)(Et,h(t′)+Et,f(t′)−Δt(t′)).=2\sum_{t=0}^{1}\sum_{t'=0}^{1}P(T=t')\left(\mathcal E^{(t')}_{t,h}+\mathcal E^{(t')}_{t,f}-\Delta^{(t')}_t\right).

By Eq. (29), Δt(t′)≥0\Delta^{(t')}_t\ge 0. Dropping these nonnegative terms from the subtraction and substituting the four inequalities above gives two transport terms in each group. Since P(T=1)+P(T=0)=1P(T=1)+P(T=0)=1, their total contribution is 4W(C,P1ϕ,P0ϕ)4W(C,P_1^\phi,P_0^\phi). Therefore,

ϵPEHE≤2(E1,h(1)+E1,f(1)+E0,h(0)+E0,f(0))+4W(C,P1ϕ,P0ϕ),\epsilon_{\mathrm{PEHE}}\le 2\left(\mathcal E^{(1)}_{1,h}+\mathcal E^{(1)}_{1,f}+\mathcal E^{(0)}_{0,h}+\mathcal E^{(0)}_{0,f}\right)+4W(C,P_1^\phi,P_0^\phi),

which establishes the population bound in Eq. (18) under the stated conditions.

Dear Reviewer 2ux9,

Thank you again for your time and valuable feedback. We would appreciate any further comments on our responses and revised manuscript. If you feel that our responses have addressed your concerns, we would be very grateful for your support for our paper during the discussions with the Area Chair.

Best regards,

The Authors

Reviewer_C53HOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method for adapting Tabular Foundation Models (TFMs) to causal effect estimation by adding a task-specific encoding layer. This layer is designed to mitigate the distributional shift between treated and control groups that arises from confounding, i.e., that patients who receive a treatment systematically differ from those who do not. The architecture also includes a gated skip connection that bypasses the foundation model, and its output is fused with the TFM output. The method is evaluated on synthetic and semi-synthetic benchmarks.

Critical strengths
  • Adapting foundation models is a promising direction for causal inference in data-scarce settings such as healthcare, and the paper makes a relevant contribution here.
  • The theoretical result, a bound on the PEHE, complements the methodological and empirical parts of the paper well.
Critical weaknesses
  • Presentation. In several places, particularly throughout Section 4, the method is presented as what feels like a chronological account of how it was developed rather than as a statement of the final model and the reasoning behind each component. For example, Eq. 9 introduces a loss, Eq. 10 extends it with a placeholder term g, and only Eq. 12 gives the loss that is actually used. This makes the method hard to follow. I recommend restructuring Section 4 to state the final model and objective first and then motivate each design choice.
  • Limited real-world evaluation. Given how directly the problem relates to real-world applications, it is unfortunate that the evaluation uses only synthetic and semi-synthetic data. With only a single semi-synthetic real-world dataset, the empirical evidence is thin. I encourage the authors to include established real-world benchmarks, such as Twins [1] or the International Stroke Trial [2].

[1] Louizos, C., Shalit, U., Mooij, J. M., Sontag, D., Zemel, R., & Welling, M. (2017). Causal Effect Inference with Deep Latent-Variable Models. Advances in Neural Information Processing Systems 30 (NIPS 2017). arXiv:1705.08821.

[2] Sandercock, P. A. G., Niewada, M., Członkowska, A., & the International Stroke Trial Collaborative Group (2011). The International Stroke Trial database. Trials, 12, 101. doi:10.1186/1745-6215-12-101.

Questions for authors
  • Could you provide an ablation of the gated skip connection? It would help to know how much of the performance comes from the foundation model versus the skip path, and how the learned gate values behave across datasets. For example, does the model largely bypass the TFM in some settings?
  • How sensitive is the method to violations of the overlap (positivity) assumption? Since the encoding layer is meant to address distributional differences between treated and control groups, results under varying degrees of limited overlap would clarify its robustness.
  • How tight is the PEHE bound in practice? Could you relate it to the empirical results, for instance by reporting the bound’s terms on the synthetic datasets, to show whether it is informative beyond motivating the design?
4 comments

Weakness 1. Presenting the final model and objective before motivating each design choice.

Answer:

We thank the reviewer for this suggestion. Our presentation proceeds from a simplified formulation to the complete model: Eq. (9) combines the frozen foundation model with Wasserstein balancing; Eq. (10) introduces the task-specific predictor with a fixed fusion weight; and Eqs. (11) and (12) introduce the learned gate and the corresponding prediction loss. Finally, Eq. (15) gives the complete training objective. We will revise Section 4 to present the final model and objective first, followed by the motivation for each component.

Weakness 2. Limited real-world evaluation and additional established benchmarks.

Answer:

We thank the reviewer for suggesting an additional real-data-derived benchmark. We have conducted supplementary experiments on Twins. Over ten experiments, tecFM-TabICL and tecFM-CausalPFN achieved Out PEHE\sqrt{\mathrm{PEHE}} values of 0.32610.3261 and 0.32550.3255, respectively, compared with 0.34120.3412 for TabICL and 0.32810.3281 for CausalPFN, as shown in the table below. We also clarify that the original evaluation includes two semi-synthetic benchmarks, IHDP and ACIC 2016. Twins provides an additional real-data-derived benchmark alongside them.

MethodOut
KNN0.5024
MLP0.3403
CFR0.3344
TabICL0.3412
DoPFN0.3352
CausalFM0.4972
CausalPFN0.3281
tecFM-TabICL0.3261
tecFM-DoPFN0.3281
tecFM-CausalFM0.3261
tecFM-CausalPFN0.3255

These supplementary experiments broaden our evaluation to the established Twins benchmark, where tecFM achieves lower Out-sample errors than the corresponding foundation models.

Question 1. Gated skip connection ablation, predictor contributions, and whether the model largely bypasses the TFM.

Answer: We thank the reviewer for this question. The reported mean gate weights show that both predictors contribute to the fused prediction rather than the model largely bypassing the TFM. Here, gtg_t weights the task-specific predictor and 1−gt1-g_t weights the foundation model, so a smaller gtg_t gives more weight to the foundation model. The first table below shows that the gate weights vary across datasets, models, and treatment conditions, while the foundation model retains at least 2727\\% of the mean weight in every reported setting. The second table compares each branch with their fusion at the same jointly trained checkpoint, without retraining. Fusion achieves lower test ϵPEHE\sqrt{\epsilon_{\mathrm{PEHE}}} than either branch alone in seven of the eight comparisons.

Mean learned gate weights. gtg_t is the weight assigned to the task-specific predictor under treatment condition tt.

DatasetICL modelgˉ0\bar g_0gˉ1\bar g_1
ACICTabICL0.0850.117
CausalPFN0.0370.043
DoPFN0.1870.216
CausalFM0.6290.703
IHDPTabICL0.2560.533
CausalPFN0.2220.489
DoPFN0.5740.674
CausalFM0.6950.730

Test ϵPEHE\sqrt{\epsilon_{\mathrm{PEHE}}} using each prediction branch alone and their learned fusion. Lower is better.

DatasetICL modelFoundation onlySmall onlyFused
ACICTabICL1.4574.0301.438
CausalPFN1.2664.3641.274
DoPFN3.30110.5352.227
CausalFM5.4293.1251.908
IHDPTabICL0.9854.0460.820
CausalPFN0.8826.3520.798
DoPFN6.1793.0960.879
CausalFM7.9822.6590.619

These results show that both the foundation model and the skip path contribute to the fused prediction, rather than the model relying solely on the skip path. Their learned fusion generally outperforms either predictor alone.

Question 2. Sensitivity to limited overlap and violations of the positivity assumption.

Answer: We thank the reviewer for this question. In our synthetic experiments, we vary α\alpha in the treatment assignment mechanism to evaluate different levels of treatment-selection bias. Larger values of α\alpha make the propensity scores more extreme and reduce overlap between the treated and control groups.

Tables 1 and 4 of the paper report the Out and Within results, respectively. Below, we reproduce the foundation model comparisons from Table 1. Across the reported settings, tecFM achieves lower Out PEHE\sqrt{\mathrm{PEHE}} than the corresponding standalone foundation model for all four backbones, providing evidence of performance under the limited-overlap settings examined.

Methodα=0.4\alpha=0.4α=0.6\alpha=0.6α=0.8\alpha=0.8α=1.0\alpha=1.0α=1.2\alpha=1.2
TabICL0.36820.32160.39230.46000.3846
tecFM-TabICL0.12070.11950.12720.12560.1249
DoPFN1.03121.09711.01711.08021.0947
tecFM-DoPFN0.16210.15910.15840.14800.2969
CausalFM2.06372.14862.11202.20072.1837
tecFM-CausalFM0.14960.13750.13560.13710.2389
CausalPFN0.24730.27780.24810.29110.2705
tecFM-CausalPFN0.12320.12120.12310.12330.1275

These results show that tecFM retains its advantage over the corresponding foundation models across the examined degrees of limited overlap, providing empirical evidence of robustness within these settings.

Question 3. Practical tightness of the PEHE bound and informativeness beyond motivating the design.

Answer: We thank the reviewer for this question. We evaluated the weighted factual prediction errors and empirical Wasserstein distance on ACIC using the same trained models and test individuals. The following table reports means across ten replications.

MethodPEHEFactual outcome prediction errors (Task-specific model)Factual outcome prediction errors (Foundation model)W^\widehat WEmpirical sum
tecFM-TabICL4.99115.84793.90010.027449.8578
tecFM-CausalPFN3.67214.08242.79350.034107.0123

The sum of the evaluated terms was approximately 1.981.98 and 1.911.91 times the mean empirical PEHE for tecFM-TabICL and tecFM-CausalPFN, respectively. This quantifies the numerical gap between the empirical evaluation of the bound's right-hand side and the actual treatment effect estimation error.

My concerns have been adressed by this comprehensive rebuttal. I thank the authors for conducting the additional experiments. I have increase my score.

Thank you for carefully considering our rebuttal and additional experiments. We are glad that your concerns have been addressed and sincerely appreciate your constructive feedback, which has helped us improve the manuscript.

Reviewer_aecUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies counterfactual prediction and treatment effect estimation. It trains an encoder to align representations for (t=0) and (t=1), a small predictor alongside a frozen foundation model, and a gate to combine their predictions.

Critical strengths

Improving foundation models for treatment effect estimation is a relevant problem. The method is evaluated across multiple foundation models and shows improvements in many settings.

Critical weaknesses

1.I do not find the motivation for this complex design convincing. It is unclear why the encoder, small predictor, and learned gate are all needed. 2.The theory provides an error bound, but does not establish when this design gives better estimation than the foundation model or simpler alternatives. 3.The paper includes ablations, but they do not fully explain the source of improvement. In particular, they do not directly support the claim that the small predictor provides better supervision to the encoder.

Questions for authors

1.How does the same encoder and small predictor perform without the foundation model? 2.Can the authors directly show that adding the small predictor helps preserve outcome-relevant information in the encoder? 3.Under what conditions should the gated combination outperform either predictor alone? Can the theory or experiments clarify this?

3 comments

Weakness 1. Necessity of the encoder, small predictor, and learned gate.

Answer: We sincerely appreciate the valuable comments provided by the reviewers.

a. Nonrandom treatment assignment can produce different covariate distributions in the treated and control groups. The trainable encoder ϕ\phi adapts the input representation to the current task by reducing this discrepancy using Wasserstein distance, while keeping the foundation model frozen.

b. Introducing a task-specific prediction branch that learns together with an encoder, allowing the representation to be usable by the base model and also supporting direct result prediction for the current task. Below is a comparison we conducted on the acic dataset.

w/o task-specific predictor (Within)w/o task-specific predictor (Out)tecFM (Within)tecFM (Out)
TabICL1.40961.46431.25141.3208
CausalPFN1.16191.34981.13471.2656

On ACIC 2016, tecFM achieves lower Within and Out$\sqrt{\epsilon_{\mathrm{PEHE}}}$than the variant without the task-specific predictor for both TabICL and CausalPFN. These results support the role of the task-specific predictor in complementing the frozen foundation model with information learned from the current task, improving treatment effect estimation rather than replacing the foundation model.

c. Fixed averaging assigns equal weights to the two predictors, whereas their relative contributions may vary across individuals and treatment conditions; we therefore introduce a learned gate to adaptively combine their predictions. Appendix C.2 compares mean aggregation with learned gating for TabICL and CausalPFN, with IHDP and ACIC 2016 results in Table 7 and simulation results in Table 8 (page 19).

Weakness 2. When the design gives better estimation than the foundation model or simpler alternatives.

Answer: We thank the reviewer for this question. Our design targets settings where confounding bias induces a distribution discrepancy between treated and control groups, limiting the foundation model's counterfactual generalization. In these settings, reducing the discrepancy while maintaining or reducing factual prediction errors tightens the upper bound, motivating representation balancing together with task-specific prediction. Tables 1 and 4 of our paper provide empirical support by comparing our method with the foundation models and other baselines under different confounding strengths.

Weakness 3 & Question 2. Better supervision and preservation of outcome-relevant information in the encoder.

Answer: On ACIC, we froze the encoders trained with or without the small predictor and learned gate, removed the original prediction branches, and trained new linear and MLP probes using factual outcomes. The probe results show that the representations learned with the small predictor and learned gate retain outcome-relevant information for both TabICL and CausalPFN.

ICL modelProbew/o small predictorwith small predictor
TabICLLinear2.0126±0.59312.0126 \pm 0.59311.6631±0.5339\mathbf{1.6631 \pm 0.5339}
TabICLMLP1.7156±0.53581.7156 \pm 0.53581.4030±0.4944\mathbf{1.4030 \pm 0.4944}
CausalPFNLinear1.4712±0.43881.4712 \pm 0.43881.4469±0.4332\mathbf{1.4469 \pm 0.4332}
CausalPFNMLP1.3161±0.4171\mathbf{1.3161 \pm 0.4171}1.3166±0.41161.3166 \pm 0.4116

The probes achieve lower mean errors in three of the four comparisons and comparable performance in the remaining comparison. Since the encoders are frozen and the original prediction branches are removed, these results provide empirical evidence that adding the small predictor and learned gate helps preserve outcome-relevant information in the encoder, beyond improving the final fused prediction.

Question 1. Performance of the same encoder and small predictor without the foundation model.

Answer: We thank the reviewer for this question. Without the foundation model, the encoder and task-specific predictor achieve a Within PEHE\sqrt{\mathrm{PEHE}} of 1.74171.7417 and an Out result of 1.80611.8061 on ACIC, as shown below.

MethodWithinOut
w/o foundation model1.74171.8061
tecFM-TabICL1.25141.3208
tecFM-DoPFN1.85582.0626
tecFM-CausalFM1.84042.0369
tecFM-CausalPFN1.13471.2656

These results show that tecFM-TabICL and tecFM-CausalPFN outperform the same encoder and small predictor on ACIC.

Question 3. Conditions under which the gated combination outperforms either predictor alone.

We thank the reviewer for this question. Gated fusion is expected to be useful when the foundation model and the task-specific predictor make complementary prediction errors. As shown in Eq. (28), the gated prediction error satisfies

(h~t−μt)2=gt(ht−μt)2+(1−gt)(ft⋆−μt)2−gt(1−gt)(ht−ft⋆)2.(\widetilde h_t-\mu_t)^2=g_t(h_t-\mu_t)^2+(1-g_t)(f_t^\star-\mu_t)^2-g_t(1-g_t)(h_t-f_t^\star)^2.

The learned gate adjusts their contributions for each sample and treatment condition. The following experimental results on ACIC further support this point.

ICL modelVariantWithinOut
TabICLFoundation model only3.09613.8793
w/o small predictor1.40961.4643
Fixed average1.46861.4907
Learned gate1.25141.3208
CausalPFNFoundation model only1.58471.5203
w/o small predictor1.16191.3498
Fixed average1.30621.3542
Learned gate1.13471.2656

Dear Reviewer aecU,

Thank you again for your time and valuable feedback. We would appreciate any further comments on our responses and revised manuscript. If you feel that our responses have addressed your concerns, we would be very grateful for your support for our paper during the discussions with the Area Chair.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary (Partition 1/2)

Dear AC,

We sincerely thank the Area Chair for their time and effort in coordinating the review process and considering our manuscript. We highly appreciate the insightful reviews, which help us improve the quality of the manuscript. We summarize the reviews and responses as follows.

1. We sincerely thank the reviewers for their positive evaluations of our work.

Reviewer aecU: "Improving foundation models for treatment effect estimation is a relevant problem. The method is evaluated across multiple foundation models and shows improvements in many settings."

Reviewer C53H: "Adapting foundation models is a promising direction for causal inference in data-scarce settings such as healthcare, and the paper makes a relevant contribution." "The theoretical result, a bound on the PEHE, complements the methodological and empirical parts of the paper well."

Reviewer 2ux9: "On originality, the question itself is timely. Causal foundation models are new, and how to adapt one to one dataset without fine-tuning has not been studied much. Using an external encoder plus a side predictor around a frozen in-context learner is a reasonable and fairly creative combination of representation balancing (in the style of CFR) and model ensembling. The over-balancing argument also gives a clear reason for the side predictor. The authors do not just add it as an extra module."

2. Summary of responses and revisions. We provide point-by-point responses to the reviewers and summarize the corresponding clarifications and supplementary experiments below.

Reviewer aecU:

  1. Motivation and necessity of the architecture (Weakness 1). We clarify the roles of the encoder, task-specific predictor, and learned gate in Sections 4.1–4.2. Supplementary ablations on ACIC examine the contribution of the task-specific predictor and compare fixed averaging with learned gating in Appendices C.4 and C.9.

  2. Conditions for improved estimation (Weakness 2). In Section 4.3, we explain how reducing the representation discrepancy while maintaining or reducing factual outcome prediction errors reduces the upper bound. Section 5.2 provides empirical comparisons under different confounding strengths.

  3. Better supervision and preservation of outcome-relevant information (Weakness 3 and Question 2). We add frozen-encoder probing experiments on ACIC in Appendix C.5. After freezing the encoders and removing the original prediction branches, newly trained linear and MLP probes evaluate the outcome-relevant information retained in the representations.

  4. Performance without the foundation model (Question 1). Appendix C.8 reports the performance of the same encoder and task-specific predictor without the foundation model. On ACIC, tecFM-TabICL and tecFM-CausalPFN achieve lower Within and Out errors than this variant.

  5. Conditions for effective gated combination (Question 3). We explain the role of complementary prediction errors using the gated squared-error decomposition in Appendix A.1. The branch ablations and aggregation comparisons in Appendices C.6 and C.9 further examine each predictor alone, fixed averaging, and learned gating.

Reviewer C53H:

  1. Presentation of the final model and objective (Weakness 1). We clarify the progression from representation balancing to task-specific prediction and learned gating. We will reorganize Section 4 to present the final model and training objective first, followed by the motivation for each component.

  2. Additional real-data-derived evaluation (Weakness 2). We add supplementary experiments on Twins alongside the original IHDP and ACIC 2016 benchmarks. Section 5.2 reports comparisons with conventional baselines and standalone foundation models, and Appendix B describes the Twins-derived data.

  3. Gated skip connection and predictor contributions (Question 1). Appendices C.6–C.7 report inference-time branch ablations and mean learned gate weights on ACIC and IHDP. The foundation model retains a substantial mean weight in every reported setting, and fusion achieves lower errors than either branch alone in seven of the eight comparisons at the same jointly trained checkpoints.

  4. Sensitivity to limited overlap and positivity (Question 2). We examine different confounding strengths by varying α\alpha in the synthetic treatment assignment mechanism. Section 5.2 and Appendix B describe these experiments and the corresponding results under limited overlap. The propensity scores remain bounded away from zero and one, so these settings retain positivity.

  5. Empirical evaluation of the PEHE bound (Question 3). Appendix C.10 reports the weighted factual outcome prediction errors, empirical Wasserstein distance, and observed PEHE on ACIC. Comparing the empirical sum with the observed PEHE quantifies the numerical gap and relates the bound expression to the experimental results.

AuthorsOpenReview ↗

Summary (Partition 2/2)

Reviewer 2ux9:

  1. Reliability of standalone baselines (Weakness 1). Appendix C.1 explains how Do-PFN and CausalFM are evaluated, including potential outcome prediction versus direct CATE prediction, the predefined IHDP data split, and categorical covariate encoding on ACIC. We also distinguish our evaluation pipeline from those used in the cited studies.

  2. Contribution beyond the task-specific predictor (Weakness 1). Appendices C.6–C.7 report branch ablations and learned gate weights. Each branch and their fusion are evaluated at the same jointly trained checkpoint without retraining. Removing the foundation branch increases the error in every reported setting, showing that the gains do not come solely from the task-specific predictor.

  3. Counterfactual-error control and theoretical assumptions (Question 1). Section 4.3 states the assumptions needed to apply Lemma 3 to the weighted losses. Appendix A.1 explicitly gives all four counterfactual-error inequalities, retaining the corresponding factual errors alongside the Wasserstein discrepancy. The pushforward identity connects the errors defined over covariates with the representation-space integrals. The weighted-loss assumptions also clarify why an invertible encoder is not required under the stated conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.