- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes test-time context augmentation for pretrained causal foundation models (CFMs) such as Do-PFN, CausalPFN and Graphs4CFM to estimate causal effects without retraining. The extra rows are added to the observational context without source labels. They are either real RCT samples, or pseudo-interventional samples drawn under , , from an auxiliary additive-noise SCM. That SCM is fitted to under a known DAG, or under a DAG built from a causal order. They use an idealized Bayesian update that treats every row as observational. Eq 14 is used for Bayesian posterior update. This is Bayes' rule applied to a two-set partition. The paper reports that augmentation brings graph-free Graphs4CFM close to its graph-conditioned version. It also improves Do-PFN on all four semi-synthetic benchmarks and CausalPFN on three.
Critical strengths
-
The proposed interface is simple and training-free that works with any model.
-
Do-PFN improves on all four semi-synthetic benchmarks at . Real RCT rows sharply improve CausalPFN in the back-door benchmark. The bivariate example and the probes suggest that the model's predictions respond to knowledge injected through the context.
Critical weaknesses
- The central question is whether routing the auxiliary SCM's samples through a CFM beats using the SCM directly, or a standard estimator with the same knowledge. However it's not clear from the empirical evidence. In table 2, the SCM alone nearly matches the augmented CFM. In table 3, the ensemble baseline is missing.
- The analysis does not explain when or why augmentation works, and the mechanistic evidence lacks controls.
- is sensitive but not sure how to select it. Lack of tests on robustness to wrong knowledge.
- The backdoor benchmark may have weak conditional treatment overlap because the latent treatment noise is small, although the actual propensity-score distribution is not reported. Constraining the marginal treatment proportion to 10–90% does not establish adequate conditional overlap. Therefore it is unclear how much of the improvement reflects extrapolation induced by the auxiliary SCM, and whether similar gains persist under moderate overlap.
Questions for authors
- Tables 3 and 8 report different values for the same configurations (e.g., DoWhy SCM PSID ATE 0.142 vs. 0.105 and Do-PFN+Aug at PSID ATE 0.202 vs. 0.190). Which are correct? Were different seeds or splits used?
- In Fig. 2a, what are , and ? Does the shift toward the prediction persist when equals the observational marginal of ? In that case an ideal Bayesian posterior should not change.
- In the probe training set in A.6.1, is the target for mixed obs/int contexts the original DAG or ? How large is the change in Fig. 2b under a control that adds the same number of rows with randomly permuted?
- What propensity-score distribution does the back-door benchmark have given for ?
- On Causal Chamber, the graph used for the auxiliary SCM and for Graphs4CFM in A.3 differs from the ground truth in Fig. 3. It has current_ls_raw but not green . Was this intended? How do the Chamber results change with the Fig. 3 graph?
- How should a practitioner choose without interventional validation data? Can you give a selection rule and evaluate it with different sample sizes?
- Can the authors evaluate robustness to controlled errors in the supplied structural knowledge, such as incorrect order relations or omitted confounder-to-outcome edges?
4 comments
Response to Reviewer nvht (2/2)
W4 + Q4. Conditional overlap in the back-door benchmark
You are right: the 10–90% marginal constraint does not ensure conditional overlap. Treatment is T = 1[f_T(x) + ε < κ] with σ_ε = 0.05, so e(x) = Φ((κ − f_T(x)) / 0.05), and the propensity score is bimodal. Exact distribution over the 300 datasets (240,000 units):
| e(x) | < 0.01 | 0.01–0.05 | 0.05–0.95 | 0.95–0.99 | > 0.99 |
|---|---|---|---|---|---|
| Share of units | 44.7% | 1.9% | 9.7% | 1.9% | 41.9% |
Per dataset, the median share of units with e ∈ (0.05, 0.95) is 0.081.
To test whether the gains depend on this, we keep the covariates, the potential outcomes and the context / test split fixed, and redraw only the treatment assignment as Bernoulli(sigmoid(α z(x))), where z(x) is the treatment latent variable, centered at the original threshold and standardized. The overlap share rises from 0.08 to 0.41 (α = 4) and 0.85 (α = 1.5). Mean PEHE, 640 context rows, 300 datasets:
| Share with e ∈ (0.05, 0.95) | CausalPFN | SCM | + Int Aug | + CF Aug | Int beats baseline | CF beats baseline | CF beats SCM |
|---|---|---|---|---|---|---|---|
| 0.08 (original) | .087 | .039 | .039 | .036 | 93% | 92% | 81% |
| 0.41 | .056 | .026 | .028 | .024 | 92% | 96% | 73% |
| 0.85 | .029 | .016 | .020 | .015 | 93% | 98% | 71% |
The absolute gain shrinks as overlap improves (Int Aug: .048 → .009), so most of the gain on the original benchmark comes from the SCM supplying the poorly supported treatment arm. Nevertheless, at every overlap level, augmentation still beats the baseline on at least 92% of the datasets. We will add the propensity-score histogram and this ablation to Appendix A.3.
Q1. Inconsistent numbers between Table 3 and Table 8
The seeds and splits were not changed; the cause is the package version: the PSID and CPS experiments in Table 3 used scikit-learn 1.5.2, while the ablations and the other experiments used 1.7.2. The values in Table 8 are the correct ones (DoWhy PSID ATE 0.105, Do-PFN + Aug 0.190). Table 3 has been updated with 1.7.2, and the conclusions are unchanged. This oversight was not intentional: the PSID and CPS experiments were the earliest ones and were run on a different server.
Q5. Graph used for Causal Chamber
The deviation from the true graph was not intended. The setting of this experiment assumes a known causal order, uses PC-simple to discover a DAG under that order, and then fits the auxiliary SCM on it; the graph in A.3 is the output of PC-simple. Results with the true graph of Fig. 3 (PEHE / relative ATE error, mean ± SE, 20 seeds):
| Method | PC-simple graph (Table 3 setting) | Fig. 3 true graph |
|---|---|---|
| CausalPFN | 91.1 ± 3.1 / .211 ± .013 | 91.1 ± 3.1 / .211 ± .013 |
| DoWhy (SCM) | 65.0 ± 3.8 / .083 ± .012 | 77.7 ± 4.4 / .161 ± .016 |
| Ensemble (⅔ CausalPFN + ⅓ SCM) | 80.0 ± 3.2 / .166 ± .012 | 85.8 ± 3.4 / .195 ± .014 |
| CausalPFN + Int Aug (r = 0.5) | 61.2 ± 3.8 / .087 ± .013 | 78.9 ± 4.0 / .156 ± .016 |
Augmentation still outperforms CausalPFN and the ensemble, and is on par with the SCM (the gap is within the standard error). The revision will state the source of the graph in A.3 and report results for both graphs.
Thank you for the clarifications and additional experiments. They address most of my concerns on empirical results and strengthen the evidence for the method’s practice value.
My main concern remains the limited conceptual contribution. The analysis does not sufficiently characterize when augmentation should help, and the proposed mechanistic explanation remains a hypothesis. In practice, there is no validated procedure for assessing the reliability of generated samples or selecting , while some structural errors substantially degrade performance. The additional gains over a better-specified auxiliary SCM also appear to be modest.
Although the empirical support is now stronger, my concerns about the contribution’s significance and systematic applicability remain. I will therefore retain my ratings.
Thank you for the follow-up and for acknowledging the strengthened empirical evidence. We briefly summarize our core contributions and add one point on r.
- A Bayesian analysis of how the model's posterior changes when unlabeled interventional rows are added to the context, which explains why such rows can improve prediction.
- A method for injecting domain knowledge into pretrained CFMs to improve prediction, with a hypothesis-based mechanistic explanation. Our probing experiments support the claim that structural information is injected.
- During the rebuttal, we added evidence and clarified our explanation: when the CFM is effective (Graphs4CFM within its prior; CausalPFN under back-door), context augmentation can improve performance and can exceed the auxiliary SCM or the CFM–SCM ensemble; and under SCM misspecification it either still improves the CFM's prediction (confounder placed after T) or reduces the bias caused by the wrong SCM (confounder placed after T and Y).
On r. Augmentation at ratio r does not behave like a prediction ensemble with the matching weight. For example, on Causal Chamber the DoWhy SCM is far stronger than CausalPFN (PEHE 65.0 vs 91.1), yet augmentation at r = 0.5 reaches a PEHE of 61.2, far below the matching ⅔–⅓ ensemble (80.0) and even below the SCM itself. A relatively small ratio, r = 0.5, already improves prediction in most of our settings and limits the bias caused by a misspecified SCM, so we recommend it as the default.
Response to Reviewer nvht (1/2)
We thank the reviewer for the detailed review.
W1. Does routing through the CFM beat using the SCM directly; ensemble baseline missing in Table 3
See G2. The ensemble baseline for Table 3 has been added (Table C): the PEHE of augmentation is lower than both the SCM and the ensemble on all four datasets. In Table 2 the SCM is close to augmentation because the SCM there uses the ground-truth DAG; the two separate when the structure or the mechanisms are wrong (Tables A, B).
W2 + Q2 + Q3. Mechanistic analysis and controls
The mechanistic hypothesis and the limits of its evidence are given in G3 and o3iZ W2. The two controls you ask for are as follows.
Q2 (Fig. 2a). ρ is the uniform distribution on the [min, max] range of the observed T, r = 1.0, N = 256. When ρ is the observational marginal of T (Gaussian), the shift no longer appears, consistent with the ideal Bayesian posterior. The shift in Fig. 2a comes from ρ differing from the observational marginal: a uniform ρ makes the marginal of T in the mixed context non-Gaussian, while Y given T remains linear with Gaussian noise, so the direction becomes distinguishable. We will state these settings and this boundary case in the main text.
Q3 (probe). In training, the target for mixed contexts is the original DAG. Control experiment: we add the same number of observational rows with T assigned at random (0 / 1 at a 1:1 ratio), without regenerating the other columns. Probe gains:
The control rows retain most of the *→T gain, which shows that this part comes from T being randomized. The T→X gain weakens and the T→Y gain disappears: the information from T to its downstream nodes has to be provided by pseudo-interventional rows.
W3 + Q6 + Q7. Choosing r; robustness to wrong knowledge
Choosing r. r is a trade-off that reflects how much the auxiliary SCM is trusted. We recommend r = 0.5 by default, which works well in most cases; use r = 1 when the structural and mechanism assumptions are trusted. When the SCM is severely wrong, a larger r causes more damage. Without interventional validation data, we currently have no data-driven selection rule, and we will state this as a limitation.
Wrong knowledge. See Table A. With wrong order relations (rev: 1 confounder placed after T; revall: all confounders placed after T), the SCM's PEHE rises to .257 (.127 – .730 for revall), whereas augmentation gives .041 (.024 – .079), still better than the baseline. With omitted confounder-to-outcome edges (cpost), augmentation is harmful: .087 → .164 (CF) / .251 (Int), but still better than the SCM's .450.
(W4 + Q4, Q1 and Q5 are answered in part 2/2.)