acceptodds
Under review as a conference paper at ICLR 2027

Context Augmentation for Causal Foundation Models with Interventional Data and Domain Knowledge

Abstract

Causal effect estimation typically requires expert knowledge to select an appropriate estimator. Recently proposed causal foundation models (CFMs) amortize the prediction of interventional distributions from observational contexts, showing promise for automatic causal effect estimation. However, observational data alone may be insufficient to identify interventional distributions: the model's implicit posterior may spread across multiple causal worlds that are observationally equivalent but imply different interventional distributions. In practice, randomized trial data or structural domain knowledge are sometimes available, yet existing CFMs either provide no interface for such information or require architectural changes and retraining. We propose test-time context augmentation, which supplies this information to a pretrained CFM through additional context samples without retraining. We first consider real interventional samples added without labels distinguishing them from observational samples. Under an idealized Bayesian analysis, we provide a posterior-odds condition under which these samples increase posterior mass on the class of structural causal models (SCMs) that induce the true interventional distribution. When only observational data are available, we fit an auxiliary SCM constrained by domain knowledge and use it to generate pseudo-interventional samples. The framework does not require the ground-truth DAG, and partial structural knowledge suffices as long as it identifies the interventional distribution. We instantiate it with a fully specified DAG and a more accessible causal order. Experiments show that context augmentation breaks observational symmetry in a non-identifiable bivariate setting and strengthens CFM representations of the local causal structure around treatment. Context augmentation also enables a graph-free CFM to match its graph-conditioned counterpart, and improves Do-PFN and CausalPFN on synthetic and semi-synthetic benchmarks.

Primary area:
causal reasoning
Keywords:
causal foundation models, treatment effect estimation, in-context learning, causal inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nvhtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes test-time context augmentation for pretrained causal foundation models (CFMs) such as Do-PFN, CausalPFN and Graphs4CFM to estimate causal effects without retraining. The extra rows are added to the observational context without source labels. They are either real RCT samples, or pseudo-interventional samples drawn under do(t)do(t), t∼ρt\sim\rho, from an auxiliary additive-noise SCM. That SCM is fitted to DobsD_{obs} under a known DAG, or under a DAG built from a causal order. They use an idealized Bayesian update that treats every row as observational. Eq 14 is used for Bayesian posterior update. This is Bayes' rule applied to a two-set partition. The paper reports that augmentation brings graph-free Graphs4CFM close to its graph-conditioned version. It also improves Do-PFN on all four semi-synthetic benchmarks and CausalPFN on three.

Critical strengths
  1. The proposed interface is simple and training-free that works with any model.

  2. Do-PFN improves on all four semi-synthetic benchmarks at r=0.5r = 0.5. Real RCT rows sharply improve CausalPFN in the back-door benchmark. The bivariate example and the probes suggest that the model's predictions respond to knowledge injected through the context.

Critical weaknesses
  1. The central question is whether routing the auxiliary SCM's samples through a CFM beats using the SCM directly, or a standard estimator with the same knowledge. However it's not clear from the empirical evidence. In table 2, the SCM alone nearly matches the augmented CFM. In table 3, the ensemble baseline is missing.
  2. The analysis does not explain when or why augmentation works, and the mechanistic evidence lacks controls.
  3. rr is sensitive but not sure how to select it. Lack of tests on robustness to wrong knowledge.
  4. The backdoor benchmark may have weak conditional treatment overlap because the latent treatment noise is small, although the actual propensity-score distribution is not reported. Constraining the marginal treatment proportion to 10–90% does not establish adequate conditional overlap. Therefore it is unclear how much of the improvement reflects extrapolation induced by the auxiliary SCM, and whether similar gains persist under moderate overlap.
Questions for authors
  1. Tables 3 and 8 report different values for the same configurations (e.g., DoWhy SCM PSID ATE 0.142 vs. 0.105 and Do-PFN+Aug at r=0.5r=0.5 PSID ATE 0.202 vs. 0.190). Which are correct? Were different seeds or splits used?
  2. In Fig. 2a, what are ρ\rho, rr and NN? Does the shift toward the T→YT\to Y prediction persist when ρ\rho equals the observational marginal of TT? In that case an ideal Bayesian posterior should not change.
  3. In the probe training set in A.6.1, is the target for mixed obs/int contexts the original DAG or GTˉG_{\bar T}? How large is the change in Fig. 2b under a control that adds the same number of rows with TT randomly permuted?
  4. What propensity-score distribution does the back-door benchmark have given σε=0.05\sigma_\varepsilon=0.05 for T∗T^\ast?
  5. On Causal Chamber, the graph used for the auxiliary SCM and for Graphs4CFM in A.3 differs from the ground truth in Fig. 3. It has current_ls_raw →Y\to Y but not green →Y\to Y. Was this intended? How do the Chamber results change with the Fig. 3 graph?
  6. How should a practitioner choose rr without interventional validation data? Can you give a selection rule and evaluate it with different sample sizes?
  7. Can the authors evaluate robustness to controlled errors in the supplied structural knowledge, such as incorrect order relations or omitted confounder-to-outcome edges?
4 comments
AuthorsOpenReview ↗

Response to Reviewer nvht (1/2)

We thank the reviewer for the detailed review.

W1. Does routing through the CFM beat using the SCM directly; ensemble baseline missing in Table 3

See G2. The ensemble baseline for Table 3 has been added (Table C): the PEHE of augmentation is lower than both the SCM and the ensemble on all four datasets. In Table 2 the SCM is close to augmentation because the SCM there uses the ground-truth DAG; the two separate when the structure or the mechanisms are wrong (Tables A, B).

W2 + Q2 + Q3. Mechanistic analysis and controls

The mechanistic hypothesis and the limits of its evidence are given in G3 and o3iZ W2. The two controls you ask for are as follows.

Q2 (Fig. 2a). ρ is the uniform distribution on the [min, max] range of the observed T, r = 1.0, N = 256. When ρ is the observational marginal of T (Gaussian), the shift no longer appears, consistent with the ideal Bayesian posterior. The shift in Fig. 2a comes from ρ differing from the observational marginal: a uniform ρ makes the marginal of T in the mixed context non-Gaussian, while Y given T remains linear with Gaussian noise, so the direction becomes distinguishable. We will state these settings and this boundary case in the main text.

Q3 (probe). In training, the target for mixed contexts is the original DAG. Control experiment: we add the same number of observational rows with T assigned at random (0 / 1 at a 1:1 ratio), without regenerating the other columns. Probe gains:

Edge type*→TT→XT→Y
Random-T control0.110.0840
DAG Aug (Fig. 2b)0.130.1510.02

The control rows retain most of the *→T gain, which shows that this part comes from T being randomized. The T→X gain weakens and the T→Y gain disappears: the information from T to its downstream nodes has to be provided by pseudo-interventional rows.

W3 + Q6 + Q7. Choosing r; robustness to wrong knowledge

Choosing r. r is a trade-off that reflects how much the auxiliary SCM is trusted. We recommend r = 0.5 by default, which works well in most cases; use r = 1 when the structural and mechanism assumptions are trusted. When the SCM is severely wrong, a larger r causes more damage. Without interventional validation data, we currently have no data-driven selection rule, and we will state this as a limitation.

Wrong knowledge. See Table A. With wrong order relations (rev: 1 confounder placed after T; revall: all confounders placed after T), the SCM's PEHE rises to .257 (.127 – .730 for revall), whereas augmentation gives .041 (.024 – .079), still better than the baseline. With omitted confounder-to-outcome edges (cpost), augmentation is harmful: .087 → .164 (CF) / .251 (Int), but still better than the SCM's .450.

(W4 + Q4, Q1 and Q5 are answered in part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer nvht (2/2)

W4 + Q4. Conditional overlap in the back-door benchmark

You are right: the 10–90% marginal constraint does not ensure conditional overlap. Treatment is T = 1[f_T(x) + ε < κ] with σ_ε = 0.05, so e(x) = Φ((κ − f_T(x)) / 0.05), and the propensity score is bimodal. Exact distribution over the 300 datasets (240,000 units):

e(x)< 0.010.01–0.050.05–0.950.95–0.99> 0.99
Share of units44.7%1.9%9.7%1.9%41.9%

Per dataset, the median share of units with e ∈ (0.05, 0.95) is 0.081.

To test whether the gains depend on this, we keep the covariates, the potential outcomes and the context / test split fixed, and redraw only the treatment assignment as Bernoulli(sigmoid(α z(x))), where z(x) is the treatment latent variable, centered at the original threshold and standardized. The overlap share rises from 0.08 to 0.41 (α = 4) and 0.85 (α = 1.5). Mean PEHE, 640 context rows, 300 datasets:

Share with e ∈ (0.05, 0.95)CausalPFNSCM+ Int Aug+ CF AugInt beats baselineCF beats baselineCF beats SCM
0.08 (original).087.039.039.03693%92%81%
0.41.056.026.028.02492%96%73%
0.85.029.016.020.01593%98%71%

The absolute gain shrinks as overlap improves (Int Aug: .048 → .009), so most of the gain on the original benchmark comes from the SCM supplying the poorly supported treatment arm. Nevertheless, at every overlap level, augmentation still beats the baseline on at least 92% of the datasets. We will add the propensity-score histogram and this ablation to Appendix A.3.

Q1. Inconsistent numbers between Table 3 and Table 8

The seeds and splits were not changed; the cause is the package version: the PSID and CPS experiments in Table 3 used scikit-learn 1.5.2, while the ablations and the other experiments used 1.7.2. The values in Table 8 are the correct ones (DoWhy PSID ATE 0.105, Do-PFN + Aug 0.190). Table 3 has been updated with 1.7.2, and the conclusions are unchanged. This oversight was not intentional: the PSID and CPS experiments were the earliest ones and were run on a different server.

Q5. Graph used for Causal Chamber

The deviation from the true graph was not intended. The setting of this experiment assumes a known causal order, uses PC-simple to discover a DAG under that order, and then fits the auxiliary SCM on it; the graph in A.3 is the output of PC-simple. Results with the true graph of Fig. 3 (PEHE / relative ATE error, mean ± SE, 20 seeds):

MethodPC-simple graph (Table 3 setting)Fig. 3 true graph
CausalPFN91.1 ± 3.1 / .211 ± .01391.1 ± 3.1 / .211 ± .013
DoWhy (SCM)65.0 ± 3.8 / .083 ± .01277.7 ± 4.4 / .161 ± .016
Ensemble (⅔ CausalPFN + ⅓ SCM)80.0 ± 3.2 / .166 ± .01285.8 ± 3.4 / .195 ± .014
CausalPFN + Int Aug (r = 0.5)61.2 ± 3.8 / .087 ± .01378.9 ± 4.0 / .156 ± .016

Augmentation still outperforms CausalPFN and the ensemble, and is on par with the SCM (the gap is within the standard error). The revision will state the source of the graph in A.3 and report results for both graphs.

Thank you for the clarifications and additional experiments. They address most of my concerns on empirical results and strengthen the evidence for the method’s practice value.

My main concern remains the limited conceptual contribution. The analysis does not sufficiently characterize when augmentation should help, and the proposed mechanistic explanation remains a hypothesis. In practice, there is no validated procedure for assessing the reliability of generated samples or selecting rr, while some structural errors substantially degrade performance. The additional gains over a better-specified auxiliary SCM also appear to be modest.

Although the empirical support is now stronger, my concerns about the contribution’s significance and systematic applicability remain. I will therefore retain my ratings.

Thank you for the follow-up and for acknowledging the strengthened empirical evidence. We briefly summarize our core contributions and add one point on r.

  1. A Bayesian analysis of how the model's posterior changes when unlabeled interventional rows are added to the context, which explains why such rows can improve prediction.
  2. A method for injecting domain knowledge into pretrained CFMs to improve prediction, with a hypothesis-based mechanistic explanation. Our probing experiments support the claim that structural information is injected.
  3. During the rebuttal, we added evidence and clarified our explanation: when the CFM is effective (Graphs4CFM within its prior; CausalPFN under back-door), context augmentation can improve performance and can exceed the auxiliary SCM or the CFM–SCM ensemble; and under SCM misspecification it either still improves the CFM's prediction (confounder placed after T) or reduces the bias caused by the wrong SCM (confounder placed after T and Y).

On r. Augmentation at ratio r does not behave like a prediction ensemble with the matching weight. For example, on Causal Chamber the DoWhy SCM is far stronger than CausalPFN (PEHE 65.0 vs 91.1), yet augmentation at r = 0.5 reaches a PEHE of 61.2, far below the matching ⅔–⅓ ensemble (80.0) and even below the SCM itself. A relatively small ratio, r = 0.5, already improves prediction in most of our settings and limits the bias caused by a misspecified SCM, so we recommend it as the default.

Reviewer_37CGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes test-time context augmentation for pretrained causal foundation models. Extra rows are appended to the model's context. They are either real RCT samples or pseudo-interventional samples drawn from an auxiliary SCM fitted with a known DAG or causal order, and the ratio r controls how many are added. A Bayesian argument shows that, provided the class-averaged likelihood condition holds, unlabeled RCT rows shift posterior mass toward SCMs that have the correct interventional distribution. Experiments apply the method to Do-PFN, CausalPFN and Graphs4CFM on synthetic back-door benchmarks and on Sales, Causal Chamber, PSID and CPS, with DoWhy and an ensemble as baselines.

Critical strengths
  1. The method is simple, needs no retraining, and works with already released models.
  2. The explanation of why unlabeled RCT rows can help is clear and intuitive.
  3. Including the auxiliary SCM (DoWhy) and an ensemble as baselines is the right design, because it isolates what the CFM adds beyond the fitted SCM.
Critical weaknesses
  1. My main concern: if you have already fitted an SCM that identifies the effect, why not use it directly? The results are weak: in Table 2, the augmented CausalPFN basically matches DoWhy. In Table 3, augmented models are best in only 3 of the 8 columns. DoWhy has the best Sales ATE, T-learner wins PSID PEHE and Chamber ATE, DML wins CPS ATE, and the non-augmented CausalPFN wins CPS PEHE. Where augmentation beats both the CFM and the SCM, the margins are often within the reported uncertainty.
  2. The theory amounts to Bayes' rule applied to the context. The condition is only argued heuristically, since the class-averaged likelihoods are acknowledged to be intractable. The pseudo-interventional case, which is the actual method, is not analyzed beyond an informal discussion of sources of error.
  3. Results are sensitive to r.
  4. Several experimental issues make the evidence weaker than presented: Tables 3 and 8 report different numbers for the same r = 0.5 runs. For example, the DoWhy PSID ATE is 0.142 vs 0.105, and the CausalPFN+Aug PSID ATE is 0.080 vs 0.074.
Questions for authors

A.3 says the PSID/CPS auxiliary SCMs "fully connect variables between adjacent levels". With the order …, Y's only parent would be T, so the SCM would not adjust for confounding. If so, how does DoWhy reach a PSID ATE error of 0.142? What are the actual parent sets of T and Y? Other questions see Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 37CG (1/2)

We thank the reviewer for the careful reading and the detailed questions.

W1. Why not use the SCM directly; weak results

Context augmentation brings practical gains on the prerequisite that the CFM itself is effective on the data (in-prior data for a general-purpose model, or the appropriate setting for a setting-specific model). In this regime, augmentation can outperform the CFM, as well as the auxiliary SCM or the ensemble of the two (see G1, G2). Regarding the specific results you point out:

  • What is being compared: our question is whether structural knowledge can improve a CFM without retraining, so the relevant comparison is among the CFM, the SCM, the ensemble and augmentation under the same knowledge. We do not claim to beat every specialized estimator in every column.
  • On the four datasets where CausalPFN is applicable, the PEHE of augmentation is never higher than that of the SCM: back-door synthetic .036 vs .039, Chamber 61.2 vs 65.0, PSID 13.96k vs 18.49k, CPS 9.43k vs 11.24k; on the semi-synthetic datasets it is also lower than the newly added ensemble baseline (Table C).
  • We agree: when the SCM is correctly specified, the gap is small and often within the uncertainty (Table 2, Chamber). The value of augmentation lies mainly in the case where the SCM is misspecified: under a wrong order the SCM's PEHE is .257 and augmentation gives .041 (Table A); under linear mechanism misspecification the SCM is at +153% and augmentation at +6.4% (Table B). The large gaps on PSID / CPS also come from an SCM that does not adjust for confounding (see Q1), and in that case the CATE predictions of the augmented CFM are only slightly affected by the wrong SCM.
  • Do-PFN: only Sales has a data distribution close to its prior, so augmentation performs well only on Sales, consistent with G1.

W2. Theory

  • We agree: the analysis is a posterior-odds condition under an idealized Bayesian setting; the class-averaged likelihoods are intractable, and the condition is argued only heuristically. Its role is to explain why unlabeled interventional rows can help a CFM pretrained on observational contexts.
  • The pseudo-interventional case has no formal analysis. Our explanation is that pseudo-interventional rows inject local structural information around T; see G3 and o3iZ W2. Whether CFMs can effectively infer structure implicitly on unseen data is not yet supported by sufficient evidence, and a systematic study is beyond the scope of this paper.
  • The support for the pseudo-interventional case is empirical: when the CFM itself is effective on the data, augmentation can outperform the CFM, as well as the auxiliary SCM or the ensemble of the two (G1); under structural or mechanism misspecification, augmentation is affected relatively little by the wrong auxiliary SCM (Tables A, B).

W3. Sensitivity to r

r is a trade-off that reflects how much the auxiliary SCM is trusted. r = 0.5 works well in most cases; when the SCM is severely wrong, a larger r causes more damage; when the structural and mechanism assumptions are trusted, use r = 1.

W4. Inconsistent numbers between Table 3 and Table 8

Thank you for pointing this out. The cause is a package version difference: the PSID and CPS experiments in Table 3 were run first, on one server with scikit-learn 1.5.2; the ablations and the remaining experiments were run later, on another server with 1.7.2. We have rerun them with 1.7.2 and updated Table 3 (DoWhy PSID ATE 0.105, CausalPFN + Aug 0.074); no conclusion is affected.

(Q1 is answered in part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer 37CG (2/2)

Q1. Parent sets of the PSID / CPS auxiliary SCM

You are right. With the level order in A.3, connecting adjacent levels gives pa(T) = {re75} and pa(Y) = {T}. The auxiliary SCM does not adjust for confounding; it returns the difference in means between the treated and control groups, which is the same constant for every unit.

  • The low ATE error on PSID is a coincidence. 93.5% of PSID units are controls, so the ATE is dominated by the outcome that control units would have under treatment. The value RealCause generates for this quantity (6.6k) happens to be close to the observed mean of the treated group (6.9k), so the unadjusted difference in means is close to the truth. On CPS the two differ (7.7k vs 4.9k), and the SCM's ATE error is larger.
  • At the individual level the SCM is poor on both datasets: its PEHE is 18.5k / 11.2k, versus 14.4k / 9.19k for CausalPFN.
  • Augmentation is only slightly affected by this wrong SCM: CausalPFN + Aug has a PEHE of 13.96k / 9.43k, close to CausalPFN and far better than the SCM.

To retain the confounders, we replace the adjacent-level connections with dense connections: every level points to all nodes in all later levels. Since Y then has many parents, the outcome mechanism is fitted separately for T = 0 and T = 1, so that the treatment effect is not swamped. Results (PEHE / relative ATE error):

DatasetCausalPFNSCM (dense graph)+ Aug
PSID14.4k / 0.22213.9k / 0.10013.8k / 0.105
CPS9.19k / 0.3149.22k / 0.2819.21k / 0.277

Here the SCM's PEHE is already close to the best in Table 3, and the augmented result moves toward the SCM. The revision will report results for both graphs and correct the description in A.3.

Thank you for the detailed response. My main concern remains: the results do not show that augmentation outperforms the better of its two inputs, the base CFM and the auxiliary SCM. 1. When we have the correct SCM (dense graph): augmentation matches the SCM on PSID and CPS. Both also match the T-learner, which uses no structural knowledge. I think this is expected, since an SCM whose outcome is fitted separately for T = 0 and T = 1 is essentially a T-learner. And 2. with the misspecified SCM (your original graph): augmentation stays close to the base CausalPFN. Its advantage is only over the misspecified SCM.

For the table 3: of the three columns where augmentation is best, the PSID ATE comes from the coincidence in the misspecified SCM. With the dense graph, the CausalPFN + Aug error rises from 0.074 to 0.105. The other two are within uncertainty. The only case where augmentation clearly beats both is Table A. However, this is a single synthetic setting. The gain is also almost the same as with the correct order, so it is unclear whether it comes from structural knowledge.

My main concren (the question raised in W1/Q1 (why not use the fitted SCM directly) therefore remains unanswered, and I will keep my score.

Thank you for the follow-up. We do not claim that context augmentation always beats a correctly specified SCM or the strongest estimator. Our claims are the following.

Setting-specific CFMs with an identifiable prior (e.g., CausalPFN under back-door). Augmentation keeps the SCM's gain when the graph is right and avoids its catastrophic errors when the graph order is wrong: in Table A it gives .036 under the true graph (SCM .039, base .087) and .038 under rev, where the SCM fails (.257). Under cpost, where the SCM degrades to .450, augmentation limits the error to .164.

General-purpose CFMs (e.g., Graphs4CFM), where observational data alone do not identify the effect. On in-prior data, augmentation improves the model even when the noise distribution is misspecified (Table B).

Does augmentation beat the better of its two inputs? In paired comparisons it does in three settings.

SettingBaseSCMAugAug below the better input (paired)
Table 2, CF Aug, PEHE, 900 datasets0.1490.0950.08269% of datasets, p = 4e-24
Table 1, Int Aug, MSE, 405 datasets0.0580.0880.05662% of datasets, p = 2e-7
Chamber (PC-simple graph), CausalPFN + Int Aug, PEHE, 20 seeds91.165.061.216/20 seeds, p = 0.007
Reviewer_o3iZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how pretrained causal foundation models (CFMs) can make use of additional causal information at inference time without retraining. The proposed approach augments the observational context with interventional samples. When real interventional data are unavailable, the authors fit an auxiliary SCM using domain knowledge, such as a DAG or causal order, and generate pseudo-interventional samples.

The paper provides a Bayesian posterior-odds analysis to motivate this approach. Experiments on synthetic and semi-synthetic datasets show improvements in causal effect estimation for several pretrained CFMs, including Do-PFN and CausalPFN. The authors also use probing experiments to investigate whether the augmented context improves the models' internal representations of causal structure.

Critical strengths
  1. The paper addresses a practical limitation of existing CFMs: how to incorporate additional causal information into models that were not explicitly trained to receive it. The proposed context augmentation is simple and does not require modifying or retraining the pretrained model.

  2. The empirical observations are interesting. In particular, the improvements from interventional context augmentation, together with the probing results on causal structure, suggest that pretrained CFMs can respond to causal information supplied indirectly through their input data. I also appreciate the comparisons with direct SCM estimation and different augmentation strategies.

Critical weaknesses

1. The generality of the empirical finding is not sufficiently established.

A central message of the paper is that providing interventional data can improve the causal effect estimation of pretrained CFMs. I find this observation interesting, but the current experiments do not establish how general or reliable it is.

The synthetic evaluations largely follow the pretraining distributions of the corresponding models, and the auxiliary SCMs in the main synthetic experiments are fitted using ground-truth DAGs. The semi-synthetic experiments provide some additional evidence, but cover relatively few settings, and the improvements are not uniform. For example, increasing the augmentation ratio can substantially degrade performance on some benchmarks.

I would like to see a more systematic understanding of when this approach works. Its effectiveness may depend on the causal graph, structural mechanisms, noise distributions, intervention targets, amount of interventional data, and accuracy of the auxiliary SCM. These factors are particularly important if the paper aims to establish context augmentation as a generally useful approach for CFMs.

2. The scientific mechanism behind the improvement remains unclear.

The paper motivates its approach through observational non-identifiability: interventional data can distinguish causal explanations that are observationally equivalent, potentially improving causal effect estimation. This is a reasonable motivation, but it raises a more fundamental question about what pretrained CFMs actually learn and use.

Does the improvement arise because the model implicitly infers a better causal structure and then uses that information for effect estimation? Or can the model estimate effects directly from the augmented statistical information, without recovering the underlying causal structure?

The probing experiments are a useful step toward understanding this question. However, they mainly show that certain structural information becomes more decodable from the hidden representations. This does not establish that the model actually uses such structural information to produce its improved effect estimates.

I think this distinction matters for the scientific contribution of the paper. Without a clearer understanding of the mechanism, the work remains primarily an empirical observation that adding appropriately constructed context can sometimes improve a pretrained model's predictions.

A related concern is the role of the auxiliary SCM. Once an SCM has been fitted, it can already be used to estimate interventional effects directly. Although the paper includes an SCM baseline, the results do not yet provide a clear understanding of when the pretrained CFM offers additional value beyond that estimator, especially when the fitted SCM is imperfect.

Overall, I find the approach potentially useful, but the current evidence does not sufficiently establish either a general empirical finding or a deeper understanding of how pretrained CFMs exploit interventional information.

Questions for authors
  1. Generality of the improvement. How broadly does the benefit of interventional context augmentation hold across different causal structures, mechanisms, noise distributions, and intervention targets? In particular, can the authors characterize when additional interventional samples help and when they hurt? It would be useful to separate real interventional data from pseudo-interventional data generated by an imperfect auxiliary SCM.

  2. The role of causal structure. The probing results suggest that structural information becomes more accessible in the model's representations after augmentation. Is there evidence that this improvement in structural representation is actually responsible for the improvement in causal effect estimation? For example, how strongly are improvements in probing accuracy associated with improvements in effect estimation across individual datasets? Could the latter occur without any corresponding improvement in causal structure inference?

  3. Value beyond the auxiliary SCM. Since the fitted auxiliary SCM can already estimate the target intervention effect, what additional information or capability does the pretrained CFM contribute? The paper includes direct SCM and ensemble baselines, but can the authors further characterize when context augmentation outperforms them, particularly as the auxiliary SCM becomes less accurate?

  4. Why unlabeled context augmentation? The method intentionally mixes observational and interventional samples without indicating their sources. How much of the benefit comes from the causal information in the intervention, as opposed to changes in the treatment distribution or effective sample balance? Could a simpler rebalancing strategy explain part of the observed improvement?

1 comment

We thank the reviewer for the thoughtful and constructive comments.

W1 + Q1. Generality: when it helps and when it hurts

Real interventional rows help in all of our experiments. The effect of pseudo-interventional rows depends on the following factors:

  • Whether the CFM itself is effective: this is the prerequisite, see G1. It holds for Graphs4CFM on in-prior data and for CausalPFN under back-door; on datasets where Do-PFN performs poorly, augmentation only falls between the SCM and Do-PFN.
  • Structure: with the correct structure, all synthetic experiments improve. With a wrong structure, the SCM can incur catastrophic error; augmentation is unaffected, or affected less than the SCM (Table A).
  • Noise distribution: misspecification has little effect; the PEHE change ranges from −4.8% to −9.2%, against −8.1% for the reference (Table B).
  • Mechanism: fitting strongly nonlinear data with linear mechanisms makes augmentation harmful (+6.4%), but far less than using that SCM directly (+153%). Our experiments fit the mechanisms with TabPFN, which works well overall.
  • Augmentation ratio r: when the auxiliary SCM has large error, a larger r causes more damage; r = 0.5 is robust in most cases.
  • Intervention target: this paper only considers interventions on the treatment variable T.

W2 + Q2. Mechanism: does the improved structural representation cause the improved effect estimation

Our hypothesis is given in G3: augmentation provides local structural information around T, from which the model implicitly infers a better structure. We agree that the probes only show that context augmentation makes the hidden layers richer in structural information; they do not prove that the model uses it to improve its predictions. The current evidence is indirect:

  • General-purpose models (Do-PFN, Graphs4CFM): ablations that provide no structural information do not improve estimation, including pseudo-observational rows (Table 1) and rebalancing. Gains appear only when structural information is injected.
  • CausalPFN: its prior is itself identifiable, so gains without structure inference are possible. Increasing overlap alone improves estimation, and pseudo-observational rows also help it (Table 2).
  • Gains in structure and gains in effect estimation need not correspond one-to-one: effect estimation also depends on the mechanisms f. Real and pseudo-interventional rows yield similar probe gains but different effect gains.

We will clarify this in the main text and state the mechanism as a hypothesis rather than a conclusion.

W2 (auxiliary SCM) + Q3. Value of the CFM beyond the SCM

See G2. There are two cases:

  1. Correct structure, biased mechanisms: when the gain from structural information exceeds the mechanism bias, augmentation can outperform both the CFM and the SCM. Augmentation is also not a simple fusion of two predictions: it outperforms the ensemble on synthetic data, and the same holds for the ensemble baseline added to the semi-synthetic datasets of Table 3 (Table C).
  2. Wrong structure or mechanisms: the SCM can fail completely, while the damage to augmentation is bounded (Tables A, B). Under some structural misspecifications the SCM fails while augmentation still performs well: with 1 confounder placed after T, the SCM's PEHE is .257 and augmentation gives .041, still better than CausalPFN without augmentation (.087).

Q4. Why no source labels; is it just rebalancing

We add no labels because the goal is to reuse existing CFMs pretrained on observational contexts, which have no input interface for source labels. On where the gains come from:

  • Rebalancing: it is ineffective on all of our datasets, so it cannot explain the gains.
  • Pseudo-observational rows (generated by the same SCM, without intervention): they give no gain for Do-PFN and Graphs4CFM, so the gain comes from the intervention itself. They do help CausalPFN, possibly because they increase overlap; hence part of CausalPFN's gain comes from the SCM supplying the poorly supported treatment arm rather than from the intervention (see nvht W4).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response (1/2)

We thank the three reviewers. Three concerns are shared: when the method works, why not use the auxiliary SCM directly, and what mechanism drives the gains. We answer them here together with the new experiments; the individual responses refer to the tables below (Tables A, B, C are in part 2/2).

G1 Scope of applicability

  • Real interventional rows: they help in all of our experiments.
  • Pseudo-interventional rows: the prerequisite is that the CFM itself is effective on the data, i.e., in-prior data for a general-purpose model (Graphs4CFM), or the appropriate setting for a setting-specific model (CausalPFN under back-door). In this regime, augmentation can outperform the CFM, as well as the auxiliary SCM or the ensemble of the two.
  • When the CFM itself performs poorly (e.g., Do-PFN on datasets far from its prior), the augmented result only falls between the SCM and the CFM.

The revision will restrict our claims to this scope.

G2 Why not use the auxiliary SCM directly

An SCM relies on assumptions about structure, mechanisms and noise, and its error can be catastrophic when these are wrong. CFMs are pretrained on a broad prior over structures, mechanisms and noise, perform strongly in small-sample regimes, and may have an implicit ability to infer structure. Augmentation passes structural knowledge to the CFM as pseudo-interventional rows while the real observational context is kept, so the damage from a wrong SCM is bounded:

  • Wrong order (Table A, rev: back-door datasets, a confounder placed after T, i.e., the edge X → T of a confounder X is reversed to T → X): the SCM's PEHE is .257, augmentation gives .041, still better than CausalPFN without augmentation (.087).
  • Linear mechanism misspecification (Table B): using that SCM directly, PEHE +153%; augmentation +6.4%.
  • Noise distribution misspecification (Table B; Gaussian, uniform, skewed, variance ×0.5 / ×2): augmentation still reduces PEHE by 4.8%–9.2%.

When the SCM is correctly specified, the augmented CFM performs close to the SCM (Table 2).

G3 Mechanism behind the gains

Our hypothesis is the following. The pretraining datasets of a CFM are sampled from ψ = (G, {f_k}, p(ε)), so the model may implicitly learn to infer the generating mechanism from observational data and then predict the interventional distribution. Augmentation additionally provides local structural information around T, from which the model infers a better structure and thus improves effect estimation.

We acknowledge that the current evidence cannot prove that CFMs actually use an implicitly inferred structure; the probes only show that the hidden layers contain more structural information after augmentation. However, in the future, general-purpose models such as Do-PFN and Graphs4CFM must be able to implicitly infer the structure or the adjustment mechanism to be truly effective, and an interface that injects structural information will then be useful. Indirect evidence and controls are given in o3iZ W2 and nvht Q2, Q3.

AuthorsOpenReview ↗

Global Response (2/2): New Experiments

G4 New experiments

Table A. Structural misspecification (back-door synthetic data, CausalPFN, n = 640, 300 datasets, mean PEHE)

Graph used by the auxiliary SCMCausalPFNSCM+ Int Aug+ CF AugInt beats baselineCF beats baselineCF beats SCM
True graph.087.039.039.03693%92%81%
rev: 1 confounder placed after T.087.257.041.03892%93%97%
cpost: confounder placed after T and Y.087.450.251.1645%25%99%

Stratified by confounding strength, revall (all confounders placed after T):

Confounding strength (median bias / sd(Y))CausalPFNTrue-graph SCMTrue graph + IntTrue graph + CFrevall SCMrevall + Int
Weak (0.19).033.022.024.020.127.024
Medium (0.63).046.029.030.027.399.033
Strong (1.25).181.066.064.062.730.079

When the order is wrong (rev, revall), the SCM fails severely, while augmentation is almost unaffected and still beats the baseline. cpost removes the confounder-to-outcome dependence entirely; augmentation gets worse (.087 → .164 / .251) but remains clearly better than the SCM.

Table B. Mechanism and noise misspecification (Graphs4CFM in-prior data, n = 248; augmentation is applied to Graphs4CFM without graph conditioning, and values are the PEHE change relative to this graph-free baseline; negative means improvement)

Setting of the auxiliary SCMPEHE change
Conditioning on the true graph (no added rows)−6.1%
Reference: TabPFN mechanisms + empirical noise−8.1%
Noise: Gaussian−6.9%
Noise: uniform−9.2%
Noise: skewed−6.9%
Noise: Gaussian × 0.5−5.0%
Noise: Gaussian × 2−4.8%
Mechanism: linear+6.4%
Mechanism: linear + Gaussian noise+3.9%
Linear SCM used directly (without the CFM)+153%

Noise misspecification has little effect. The mechanisms of this prior are strongly nonlinear with heavy-tailed noise, so a linear mechanism is a severe misspecification; augmentation is then harmful, but far less so than using that SCM directly.

Table C. Ensemble baseline added to Table 3 (CausalPFN, PEHE / relative ATE error; ensemble = ⅔ CausalPFN + ⅓ SCM, matching the row ratio at r = 0.5)

DatasetCausalPFNSCMEnsemble+ Int Aug (r = 0.5)Seeds where Aug has lower PEHE than the ensemble
Chamber91.1 / .21165.0 / .08380.0 / .16661.2 / .08720 / 20
PSID (×10³)14.40 / .22218.49 / .10515.03 / .14313.96 / .07499 / 100
CPS (×10³)9.19 / .31411.24 / .4199.46 / .3439.43 / .39967 / 100
Sales (×10⁴)58.0 / .43335.2 / .04138.3 / .27534.4 / .1634 / 5

The PEHE of augmentation is lower than both the SCM and the ensemble on all four datasets. On CPS it is slightly worse than CausalPFN without augmentation; on ATE, the SCM is better on Chamber and Sales.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.