acceptodds
Under review as a conference paper at ICLR 2027

CAFE: Counterfactual Prediction via Fast Posterior Estimation

Abstract

Counterfactual prediction estimates an individual's outcome under an alternative intervention given their factual observations. Such outcomes are generally not identifiable from observational data without additional assumptions. Even within the class of fully observed additive noise models (ANMs), different causal graphs can generate the same observational distribution yet imply different individual counterfactual outcomes. Predictions based on a single estimated graph ignore this structural uncertainty. We therefore target a Bayesian counterfactual posterior predictive distribution that combines predictions from plausible SCMs. We introduce CAFE (CounterfActual Prediction via Fast Posterior Estimation), an amortized inference framework that directly approximates the Bayesian counterfactual posterior predictive distribution. We pretrain a transformer-based model on synthetic counterfactual tasks generated from a diverse prior over ANMs. Given an observational dataset, an individual's factual observations, and an intervention, CAFE approximates the corresponding posterior predictive distribution in a single forward pass. Experiments show that CAFE accurately predicts individual counterfactual outcomes in identifiable settings and approximates the posterior predictive distribution when structural uncertainty induced by observationally indistinguishable causal graphs exists. Strong performance in realistic manufacturing and viticulture settings further demonstrates its empirical robustness beyond the assumptions of the training prior.

Primary area:
causal reasoning
Keywords:
Counterfactual prediction, Bayesian inference, In-context learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AEKiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CAFE, a prior-data fitted network for individual counterfactual prediction. The method trains a transformer on synthetic data generated from fully observed additive noise SCMs with varying causal graphs. Given observational data, an individual's factual observations, and an intervention, CAFE directly predicts the Bayesian counterfactual posterior without explicitly estimating the causal graph. Experiments evaluate prediction accuracy in identifiable settings, uncertainty quantification under graph non-identifiability, and robustness on realistic simulated datasets.

Critical strengths
  • Relevant and challenging problem: The paper addresses individual-level counterfactual prediction without requiring knowledge of the underlying causal graph.

  • Strong empirical performance: CAFE demonstrates consistently competitive or superior counterfactual prediction accuracy across diverse benchmarks, including settings that violate its modeling assumptions.

Critical weaknesses
  • Limited novelty: The proposed approach closely follows existing causal PFNs (e.g., Ma et al., CausalFM; Robertson et al., Do-PFN), which already amortize Bayesian causal inference over SCM priors, including uncertainty over causal graphs. These existing frameworks could be straightforwardly adapted to individual counterfactual prediction by conditioning on the observed factual outcome YY, modifying the prediction target, and restricting the SCM prior to identifiable classes such as additive noise models. It remains unclear what fundamentally new methodological contribution CAFE provides beyond this adaptation. Additionally, these straightforward adaptations could be considered strong experimental baselines that are ommitted.

  • Insufficient discussion of related work and missing baselines: Crucial related work on causal PFNs is discussed only in the appendix. The authors primarily distinguish these approaches by their prediction targets, without explaining the technical contributions of CAFE relative to existing methods. Moreover, the main experiments do not compare against causal PFNs such as CausalFM or Do-PFN, nor with regular TabPFN.

  • Theory: The theoretical results seem largely straightforward adaptations of known results. It would be helpful if the authors could elaborate on their contribution here.

  • Assumptions. Individual counterfactual outcomes are generally not identifiable even when interventional distributions are known. CAFE circumvents this by imposing strong structural assumptions, particularly fully observed additive noise models with independent exogenous noise. Such assumptions may be unrealistic in many practical applications, limiting the applicability of the proposed framework. This is primarily a limitation of the considered problem setting rather than the specific methodology.

  • Prior dependence and uncertainty evaluation: When the causal graph is non-identifiable, the proposed counterfactual posterior depends on the assumed SCM prior rather than being uniquely determined by observational data. The non-identifiability experiment primarily evaluates whether CAFE approximates the posterior induced by its own prior, without assessing sensitivity to prior misspecification. Furthermore, experiments violating the ANM assumptions evaluate point-prediction accuracy but not the reliability of the predicted counterfactual uncertainty.

Questions for authors

It would be helpful if the authors could elaborate on their contribution. What changes conceptually beyond "conditioning on Y" and restricting the prior?

2 comments
AuthorsOpenReview ↗

Response to Reviewer AEKi

We thank the reviewer for suggesting closer comparisons with existing causal PFNs and broader evaluation of prior sensitivity and counterfactual uncertainty. These comments helped us improve the paper’s positioning and make the evaluation more comprehensive. In the revised PDF, we have expanded the related-work discussion, added foundation-model baselines, prior-sensitivity experiments, and uncertainty metrics beyond the ANM setting.


W1 & Q1: What does CAFE contribute beyond existing causal PFNs?

A: CAFE’s contribution is to develop an effective PFN-based model for individual counterfactual prediction and establish its value through extensive experiments. PFNs provide a reusable learning paradigm, but existing pretrained causal PFNs do not directly solve this challenging counterfactual prediction task. CAFE demonstrates that this paradigm can deliver accurate individual predictions in identifiable settings and capture posterior uncertainty in non-identifiable settings. Its consistent improvements over CATE-adjusted foundation-model baselines (Table 4) further support the value of developing and training a model specifically for counterfactual prediction task.


W2: Could the authors expand the discussion of causal foundation models and add comparisons with them in the main text?

Yes. We have added discussion of these related work in the Introduction and add these causal foundation models as import baselines in Section 4.4. We recognize causal foundation models, including Do-PFN and CausalFM, as important methodological contributions to the field of causal inference. They helped pioneer a new paradigm which uses pretraining to amortize Bayesian posterior inference for causal queries. They target L2 causal queries. We adapt their predictions through CATE-adjustment to incorporate our L3 task formulation. We evaluate them on binary-treatment datasets to accommodate their supported treatment settings. Table 4 shows that CAFE outperforms these baselines.

Table 4: RMSE on C-Suite and MAPE (%) on Sangiovese.

Methodlarge backdoor btmixed confoundingmixed simpsonBrixpH
CausalFM0.10 ± 0.000.51 ± 0.000.48 ± 0.002.29 ± 0.041.18 ± 0.01
Do-PFN0.09 ± 0.000.51 ± 0.000.50 ± 0.002.93 ± 0.020.79 ± 0.01
CAFE0.05 ± 0.000.41 ± 0.000.38 ± 0.001.99 ± 0.010.72 ± 0.02

W3: What does the theoretical analysis contribute beyond established results?

A: Building on established counterfactual identification theory, our analysis characterizes CAFE’s target posterior and explains the uncertainty that remains when compatible graphs imply different counterfactual outcomes. Our main contribution is a pretrained method for individual counterfactual prediction, together with extensive empirical validation of the PFN approach on this task.


W4: How do CAFE’s assumptions affect its practical applicability?

A: Counterfactual prediction necessarily relies on assumptions about the underlying data-generating process. We agree that CAFE’s fully observed ANM assumptions can be restrictive in practice. Therefore, we evaluate CAFE in real world scenarios, providing empirical evidence of its predictive robustness when these assumptions are violated (Section 4.3).


W5.1: How sensitive are CAFE’s predictions to the pretraining prior?

A: We vary the pretraining graph and noise priors while evaluating all variants on the same datasets. Performance remains broadly stable under moderate prior changes (Appendices D.7.2–D.7.3).


W5.2: Could the authors evaluate the reliability of predicted counterfactual uncertainty beyond the ANM setting?

A: Yes. We have expanded the Sangiovese evaluation to include CRPS and Coverage@90 in Section 4.4. CRPS evaluates each predicted distribution against the ground-truth counterfactual outcome, rewarding accuracy while penalizing excessive uncertainty. Coverage@90 is a uncertainty calibration method that measures the fraction of counterfactual outcomes within the predicted 5th–95th percentile interval, with a nominal target of 90%. CAFE’s coverage is closest to this target on Brix. On pH, CAFE is second only to Do-PFN.

Table 5: Uncertainty quantification on Sangiovese.

MethodBrix CRPSBrix Coverage@90pH CRPSpH Coverage@90
CausalFM0.018 ± 0.0000.966 ± 0.0010.009 ± 0.0000.760 ± 0.011
Do-PFN0.040 ± 0.0000.883 ± 0.0050.018 ± 0.0000.897 ± 0.004
CAFE0.014 ± 0.0000.896 ± 0.0020.007 ± 0.0000.969 ± 0.003

If the reviewers have any remaining questions, we would be glad to discuss and clarify them further.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns.

  • Novelty (W1): My main concern remains unresolved. CAFE appears to follow the established causal PFN paradigm, primarily changing the conditioning inputs, SCM prior, and prediction target. The authors do not clearly identify a methodological contribution beyond this straightforward adaptation.

  • Related work and baselines (W2): The additional comparisons with CausalFM, Do-PFN, and TabPFN are valuable and demonstrate strong empirical performance. However, adapting pretrained L2 models through CATE adjustment does not address whether existing causal PFN approaches could be straightforwardly retrained for L3 prediction.

  • Theory and assumptions (W3–W4): I appreciate the clarifications and acknowledgment of restrictive assumptions. However, the theoretical contributions remain largely adaptations of established results.

  • Prior sensitivity and uncertainty (W5): The additional experiments are useful. Nevertheless, some prior changes substantially affect performance, and the uncertainty evaluation shows imperfect calibration (e.g., 96.9% coverage at a nominal 90% level for pH).

Overall, the rebuttal substantially strengthens the empirical evaluation, but my central concern regarding methodological novelty remains. I therefore maintain my rating.

Reviewer_3ziBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets individual counterfactual prediction (Pearl's L3). The authors show that even in fully observed additive noise models with independent exogenous noises, observationally equivalent graphs can imply different individual counterfactuals, so predicting from a single estimated graph ignores residual structural uncertainty. Their target is instead the Bayesian counterfactual posterior predictive, a mixture of point masses, one per observationally compatible graph, weighted by graph posterior probability under a prior over ANMs. Proposition 1 shows that within the ANM class a compatible graph plus the exact observational distribution uniquely fixes the individual counterfactual, and Proposition 3 proves weak convergence of the finite-sample posterior predictive to the population version. Proposed method CAFE is a prior-data-fitted network: a 12-layer tabular transformer pretrained on synthetic tasks, namely SCMs from a hand-designed prior over DAGs with MLP mechanisms and multiple root/noise families, plus factual/counterfactual pairs sharing exogenous noise. It is trained with a 5000-bin discretized NLL on the counterfactual outcome, so a single forward pass returns a predictive distribution over YtY^t with no graph search or per-dataset fitting. Unlike priors that force a back-door structure, this one places TT and YY anywhere in the graph, so XX may be caused by treatment. Validation covers identifiable C-Suite settings, where CAFE has the lowest RMSE of seven baselines on six of seven datasets. It also covers an analytically tractable non-identifiability benchmark, six complete DAGs on (T,X,Y)(T, X, Y) sharing one Gaussian observational distribution, where posterior mean RMSE, variance RMSE and IQD generally fall with NN while DECI and DiBS misallocate the known reference posterior.

Critical strengths
  1. A precise structural-uncertainty target for L3 prediction, with an analytic benchmark for it, the first I am aware of, and a meaningful negative result. Eqs. 3–4 make explicit what should be predicted when observationally equivalent ANMs disagree, and in Section 4.2 authors turn this into a falsifiable test with an exact reference posterior. CAFE errors generally decrease with NN, which is shown in Table 2. Table 5 and Figure 4 show that two Bayesian structure learners that maintain graph posteriors still fail to preserve the correct counterfactual posterior. DiBS over-allocates mass to dominant modes and DECI concentrates most of graph mass on graph classes that cannot reproduce PVP_V. This benchmark is a durable contribution. It sets a standard for evaluating uncertainty-aware counterfactual predictors, and it separates CAFE from the L2-oriented causal PFNs it cites.

  2. An honest presentation of when amortizing over a diverse structural prior buys something. Authors show that standard CEPO estimators plus CATE adjustment (y+μ^(x,t)−μ^(x,tf)y + \hat{\mu}(x,t) - \hat{\mu}(x,t_f)) come close to CAFE in three of the four settings where Proposition 4 conditions hold with a substantial gap remaining on Large Backdoor (best adjusted 0.41 against CAFE 0.15), and stall behind in the four where they fail. The back-door-prior ablation reproduces the same pattern from the pretraining side, with the correct mechanistic explanation. The adjustment restores the individual's centered noise explicitly, while the back-door-trained model learns to infer it implicitly. A reader gets an actionable rule for when a CATE pipeline suffices and when CAFE's machinery is warranted.

Critical weaknesses
  1. The main claim is validated only on one linear-Gaussian toy. CAFE difference from prior causal PFNs is the mixture over observationally equivalent graphs. Its only ground-truth check of the non-degenerate posterior, the mixture weights that are the actual contribution, is the three-variable, linear-Gaussian, six-DAG system. In identifiable settings the only distributional evidence is shown in App. D.1, Figure 3, where the ground truth is a point mass, so any sharply concentrated density scores well there. No reference posterior is computed for any nonlinear ANM, which is the regime the method is built for.

  2. The prior that defines the posterior is assumed and untested. The target is defined by Π\Pi, but the trained model's effective prior is set by finite capacity, the B=5000B = 5000-bin discretization, and a finite set of pretraining tasks. It is never characterized. The paper reports no calibration evaluation of the predicted distributions. Full-distribution quality rests on a single IQD metric and only on the Gaussian toy. No log-score or CRPS-type check is reported. PFN pretraining priors are known not to be asymptotically overwritten by data. Melnychuk et al. (ICML 2026) show prior-induced confounding bias for Bayesian ATE estimators.

  3. Part of the out-of-assumption robustness evidence is degenerate. In Table 3 at t=30000t = 30000, CAFE and DiBS both score exactly 100.00 while the factual baseline sits at 12.15 and 6.70. Both models appear to predict a saturated outcome class. That column then tests whether a method flips to the majority label, not whether it predicts counterfactuals. As a result, the robustness case described in Section 4.3 is partly inflated, and it should rest on the t=15000t = 15000 results, where CAFE shows sensitivity to the intervention (98.05 and 97.98 against factual baselines of 92.85 and 94.05) while DiBS matches the factual baseline.

  4. The paper states that "the work most closely related to CAFE is TabPFN-CFM (Zhu et al., 2026)". Several published works are closer on the framing axis. Miller, Schölkopf & Guo (NeurIPS 2025) analyze a pretrained transformer that answers individual counterfactual queries in context, set up as a posterior-predictive computation, trained on synthetic SCM sequences, with dedicated noise abduction heads. Chen & Du (ICML 2025) prove counterfactual identifiability for bijective SCMs under a known causal order, via exogenous isomorphism.

  5. This paper does not cite Shpitser & Pearl (2008), no Tian & Pearl (2000), no Künzel et al. (2019) and no Shalit et al. (2017). The first two establish when individual counterfactuals are identifiable and what bounds exist when they are not, which is the premise of Sections 1 and 2. Künzel et al. is related to the factual-plus-CATE-adjustment construction of Proposition 4. Shalit et al. is the standard reference for individual counterfactual and ITE prediction. The positioning claim also needs relocating: Section 3.1 argues its non-back-door prior against CausalPFN back-door setting (line 203), while TabPFN-CFM appears only in Appendix, and neither argument engages this identifiability and ITE literature.

  6. The population-objective optimality statement in Section 3 is given without attribution. Lines 189–191 state that with sufficient capacity and exact optimization of the population objective the optimal predictor matches the Bayesian posterior predictive for almost every input under PΠP_\Pi. The sentence carries no citation. It is the standard PFN result, Müller et al. (ICLR 2022).

Questions for authors
  1. Can authors evaluate CAFE against a ground-truth counterfactual posterior beyond the six-DAG linear-Gaussian system, e.g., a 3–4 variable nonlinear ANM where Eq. 4 is approximated by enumerating DAGs and running MCMC over mechanisms?

  2. The target posterior is defined by Π\Pi, but the trained model's implicit prior is not inspectable. Published analysis of causal PFNs shows that, for ATE estimands, the pretraining prior is not asymptotically overwritten by data and induces bias (Melnychuk et al., 2026), established there only for finite-dimensional estimands but a natural caution here. (a) Could authors provide evidence that CAFE's learned prior tracks Π\Pi as NN grows? (b) Would a perturbed pretraining prior shift predicted mixture weights accordingly, and does the frozen model adapt at all?

  3. Please report the base rate of ProcessResult under do(t=30000)\mathrm{do}(t = 30000) and the accuracy of a constant-predictor baseline. DiBS also scores exactly 100.00±0.00100.00 \pm 0.00 in both CausalMan settings, while the factual baseline sits at 12.15 and 6.70. That suggests both models simply predict one saturated class. If so, how much should the t=30000t = 30000 columns contribute to the out-of-assumption robustness claim? CAFE's t=15000t = 15000 results already show sensitivity to the intervention, while DiBS matches the factual baseline there, so that claim does not depend on these two columns.

  4. Please confirm or correct: (i) Proposition 1 is an instance of the exogenous-isomorphism result of Chen & Du (ICML 2025), restricted to bijective additive-noise mechanisms under a known causal order, or does their additional identical-counterfactual-transport assumption make the two results incomparable? (ii) Miller et al. (NeurIPS 2025) already document the composition of amortized in-context individual counterfactual prediction. (iii) The optimality statement in Section 3, that sufficient capacity plus exact optimization of the population objective yields the Bayesian posterior predictive, restates Cor. 1.2 of Müller et al. (ICLR 2022). A short positioning against these works, beyond the TabPFN-CFM comparison, would strengthen the paper.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3ziB

We thank the reviewer for recognizing that CAFE addresses an important and challenging counterfactual prediction task and for highlighting the value of our experimental findings. Following your suggestions, we have revised the PDF to broaden the evaluation and clarify connections to prior work.


W1 & Q1: Can CAFE accurately predict non-degenerate counterfactual posteriors in nonlinear ANMs?

A: Yes. We demonstrate this with a controlled three-variable nonlinear ANM experiment. The prior covers two observationally equivalent DAGs with varying mechanism parameters, yielding an exact counterfactual posterior that generally remains non-degenerate (Appendix D.9). As shown in Table 16, CAFE’s predictions become closer to the ground-truth counterfactual posterior as the context size increases.

Table 16: IQD across context size N.

N=10N=50N=100
DiBS0.052 ± 0.0280.037 ± 0.0200.034 ± 0.019
CAFE0.050 ± 0.0490.035 ± 0.0250.031 ± 0.024

W2: How well does CAFE approximate distributions and calibrate uncertainty beyond the Gaussian setting?

A: We add CRPS and Coverage@90 evaluations on Sangiovese, whose data-generating mechanisms violate the ANM assumptions (Section 4.4). CRPS evaluates each predicted distribution against the counterfactual outcome. CAFE achieves the lowest CRPS on both Brix and pH. We assess uncertainty calibration using Coverage@90, where values closer to the nominal coverage of 0.90 indicate better calibration. CAFE is closest to this target on Brix and second closest on pH, after Do-PFN.

Brix CRPSBrix Coverage@90pH CRPSpH Coverage@90
Do-PFN0.040 ± 0.0000.883 ± 0.0050.018 ± 0.0000.897 ± 0.004
CausalFM0.018 ± 0.0000.966 ± 0.0010.009 ± 0.0000.760 ± 0.011
CAFE0.014 ± 0.0000.896 ± 0.0020.007 ± 0.0000.969 ± 0.003

Q2.a: Does CAFE track the Bayesian counterfactual posterior as the context size grows?

A: Yes. In our controlled experiment, CAFE’s predictions become closer to the Bayesian counterfactual posterior as more observations are provided. We retrain CAFE in a linear-Gaussian setting where this posterior is tractable. Table 15 shows decreasing prediction errors relative to both this posterior and its population limit (Appendix D.7.1).

Table 15: IQD of CAFE predictions relative to the Bayesian posterior and its population limit.

Context sizeBayesian posterior errorPopulation limit error
100.012 ± 0.0040.020 ± 0.002
200.010 ± 0.0020.016 ± 0.003
1000.008 ± 0.0020.011 ± 0.002

Q2.b: If the pretraining prior changes, do CAFE’s predicted mixture weights change accordingly?

A: Yes. We test this in setting of Section 4.2 by retraining CAFE under modified priors. This setting involves 6 observationally equivalent DAGs whose ground-truth posterior weights depend on the specified prior. We vary the prior using a parameter s: larger s assigns greater prior probability to graphs in which Y is a descendant of T, thereby increasing their posterior weights in this evaluation case. We train variants for each s∈1/3,1,3,6s\in\\{1/3,1,3,6\\}. Figure 6 and Table 12 (Appendix D.7.1) show that the predicted mixture weights change in accordance with the modified prior. For a frozen model, this prior change cannot be inferred from observational data alone, so the data cannot reveal how their prior weights have changed.

Table 12: IQD between predictions and the ground-truth posteriors.

Context sizes=1/3s=1s=3s=6
500.14 ± 0.050.16 ± 0.050.26 ± 0.030.30 ± 0.05
1000.10 ± 0.060.10 ± 0.090.18 ± 0.040.21 ± 0.07
10000.01 ± 0.000.02 ± 0.000.02 ± 0.000.03 ± 0.01

W3 & Q3: What does the saturated-outcome setting contribute to the robustness evaluation?

A: Outcome saturation is a practically meaningful consequence of strong intervention: the task is to predict that individuals with different factual outcomes reach the same outcome after intervention. We interpret the t=30000 results in this sense.


W4–W6 & Q4: Could the authors clarify CAFE’s relationship to prior work and provide the missing citations?

A: Yes. We have expanded the related-work discussion, added the missing citations, and revised the “most closely related” claim. We distinguish CAFE from Miller et al., which consider a fixed causal structure: CAFE predicts a prior-dependent counterfactual posterior accounting for uncertainty across observationally equivalent graphs (Appendix C.2). We also clarify connections to counterfactual identification, partial identification, and treatment-effect estimation through the requested references (Section 1, line 38; Section 2; Appendices C.1 and D.5), and explicitly attribute Müller et al. (Section 3, line 206) and Chen & Du (Section 1, line 46; Appendix A.1).

AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Thank you for your thoughtful comments and for recognizing the value of our work. We have added a nonlinear ANM benchmark, uncertainty calibration evaluations, and experiments on posterior approximation and prior sensitivity, and expanded the discussion of related work. These updates are summarized in our rebuttal and detailed in the revised manuscript. We would appreciate your thoughts on these updates and would be happy to address any remaining questions.

Thanks for the additions. W5, W6 and Q4 are settled. Four points remain open.

1. Table 16 does not test the mixture claim (W1, Q1). IQD is scale-dependent: against the D.9 truth 12δa+12δb\tfrac{1}{2}\delta_a + \tfrac{1}{2}\delta_b, a predictor w δa+(1−w) δbw\,\delta_a + (1-w)\,\delta_b scores (w−12)2∣b−a∣(w-\tfrac{1}{2})^2 |b-a|, and any point mass in [a,b][a,b] scores ∣b−a∣/4|b-a|/4. CAFE's best entry (0.0290.029 at N=200N=200) equals the point-mass score when ∣b−a∣≈0.12|b-a| \approx 0.12, and D.9 does not report the atom gap, so the table cannot separate correct weights from a collapsed prediction. Please report, on the same 30 tasks, the IQD of the factual-copy predictor δy\delta_y, which is exactly the G0G_0 atom.

2. Table 16 is parity, not an advantage. Gaps of 0.0020.002–0.0070.007 against standard deviations of 0.0140.014–0.0490.049: does a paired comparison over the 30 tasks change the ranking? If not, please call it parity.

3. Table 15 implicates the model, not the data (W2, Q2a). Approximation error exceeds finite-data error at every NN, and the ratio widens (1.31.3 at N=10N=10 to 2.72.7 at N=100N=100). Does the approximation error keep falling at N=250N = 250, 500500, 10001000, or flatten? A plateau would indicate an approximation floor.

4. Table 3 at t=30000t=30000 does not discriminate methods (W3, Q3). Your response confirms saturation: every individual reaches the same post-intervention label. A constant predictor then also scores 100%100\%, as do CAFE and DiBS. Please add the post-intervention base rate and a constant-predictor row to Table 3, and qualify "highest accuracy in all four settings" (l.459): at t=30000t=30000 CAFE ties DiBS and the constant baseline. The robustness claim should rest on t=15000t=15000.

Points 1 and 4 could move my rating. For point 2 parity wording suffices, and for point 3 reporting a plateau as a limitation is acceptable. The analytic benchmark and the DECI/DiBS negative result remain real contributions.

Reviewer_JrQNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors consider individual counterfactual prediction problem when the causal graph is unknown. The authors propose CAFE (Counterfactual prediction via Fast Posterior Estimation), an amortized inference framework to approximate the Bayesian counterfactual posterior predictive distribution. The proposed approach involves a tabular transformer pretrained on synthetic tasks generated from a prior over additive noise models. Empirically, the authors evaluate the proposed approach in identifiable settings, counterfactual uncertainty under non-identiable causal structure, and robustness.

Critical strengths
  • The considered problem is well formulated, i.e., same observational distribution, but imply different individual counterfactual outcomes.

  • The proposed approach is potentially applicable to a broader set of counterfactual estimation problems, i.e., pretraining on synthetic tasks can provide counterfactual predictions without explicit graph estimation, or any parameter updates on new datasets.

  • The authors empirically illustrate advantages of the proposed approach.

Critical weaknesses
  • The relationship between the theoretical assumptions and the implemented task generator may not be clarified well, e.g., when generated tasks satisfy the proposition? and what happens when they do not.

  • It is better to analyze the sensitivity to the training prior rigorously.

  • It is unclear about the effect of the trained discretized predictor on the posterior convergence results involving exact Bayesian inference.

  • It is better to clarify main components of the proposed framework, especially on the prior and synthetic task generation, and transformer training.

Questions for authors
  • Could the authors distinguish errors caused by finite observed data from errors in CAFE’s posterior approximation in the experiments?

  • Could the authors illustrate the sensitivity to the prior and unequal posterior weights? graph structures? noisy priors?

  • Could the authors clarify the number of synthetic tasks, the pretrained transformer? It is better to provide rigorous details on the pretrained checkpoints and evaluated tasks? How does the pretrained checkpoint affect the performances? are the performances sensitive across independent pretraining runs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer JrQN

We thank the reviewer for the valuable suggestions on clarifying our assumptions and pretraining procedure and strengthening the empirical analysis. We have revised the PDF accordingly to address these concerns.


W1: Relationship between the theoretical assumptions and the pre-training task generator.

A: We apologize that this relationship was not clearly explained. The task generator samples from the general ANM class, but generated tasks may not satisfy all conditions (i)–(iii) of Proposition 2. Our theoretical analysis distinguishes the following situations:

  • Full identifiability. Under Assumption 1 and conditions (i)–(iii), Proposition 2 guarantees counterfactual point identification.
  • Graph non-identifiability. Under Assumption 1, when condition (i) is violated and the causal graph is not uniquely identifiable, Equation (3) accounts for graph uncertainty in the counterfactual posterior.
  • Violation of Assumption 1. If the model is not an ANM, our analysis provides no guarantee.

Our task generator covers situations 1 and 2. In our evaluation, CAFE achieves accurate predictions in situation 1 (Section 4.1), approximates the counterfactual posterior well in situation 2 (Section 4.2), and performs robustly when Assumption 1 may not hold (Section 4.3).


W2 & Q2: Can CAFE recover unequal posterior weights when the training prior changes? How sensitive is its performance to graph-structure and noise priors?

A: CAFE can captures unequal graph posterior weights, is robust to moderate changes in structural and noise-strength priors, and benefits from noise diversity.

  • Unequal posterior weights. We extend the experiment in Section 4.2 to test whether CAFE can recover counterfactual posteriors with unequal mixture weights. This experiment involves 6 observationally equivalent DAGs whose ground-truth posterior weights depend on the prior. We vary the prior using a parameter s: larger s assigns greater prior probability to graphs in which Y is a descendant of T, thus increasing their posterior weights in this evaluation case. We train variants for each s∈1/3,1,3,6s\in\\{1/3,1,3,6\\}. Table 12 and Figure 6 show that CAFE accurately approximates the corresponding counterfactual posterior (Appendix D.7.1).
  • Graph-structure priors. We also examine whether changing the pretraining graph prior affects CAFE’s performance on identifiable tasks. We evaluate the four models trained above with s∈1/3,1,3,6s\in\\{1/3,1,3,6\\} on C-Suite. Table 13 in Appendix D.7.2 shows broadly stable performance for s∈1,3,6s\in \\{1,3,6\\}.
  • Noise priors. We also vary the pretraining noise prior and evaluate its effect on C-Suite. Table 14 in Appendix D.7.3 shows comparable performance among models pretrained with low, uniform, and high-SNR prior. Separately, replacing the mixture of noise families with Gaussian noise alone degrades performance on some datasets.

Table 12: IQD between predictions and the ground-truth posteriors.

Context sizes=1/3s=1s=3s=6
500.14 ± 0.050.16 ± 0.050.26 ± 0.030.30 ± 0.05
1000.10 ± 0.060.10 ± 0.090.18 ± 0.040.21 ± 0.07
10000.01 ± 0.000.02 ± 0.000.02 ± 0.000.03 ± 0.01

Table 13 & 14: RMSE under different pretraining graph and noise priors.

fork nonlin gausslarge backdoorweak arrows
Default (s=1, high SNR)0.01 ± 0.000.15 ± 0.020.14 ± 0.00
Graph: s=1/30.00 ± 0.000.35 ± 0.000.39 ± 0.02
Graph: s=30.01 ± 0.000.13 ± 0.040.10 ± 0.01
Graph: s=60.02 ± 0.000.20 ± 0.020.09 ± 0.01
Noise: low SNR0.03 ± 0.000.15 ± 0.010.10 ± 0.00
Noise: uniform SNR0.02 ± 0.000.18 ± 0.010.15 ± 0.01
Noise: Gauss0.00 ± 0.000.64 ± 0.010.87 ± 0.02

W3: How does CAFE’s discretized predictor relate to posterior convergence?

A: We analyze the effect of discretization by varying the number of output bins. Empirically, predictive accuracy remains broadly stable across the bin counts (Appendix D.10).


Q1: Can finite-data error be distinguished from CAFE’s posterior approximation error?

A: Yes. We retrain CAFE under a controlled prior with a tractable exact finite-data posterior to distinguish finite-data error (finite-data vs population posterior) from approximation error (CAFE vs finite-data posterior). Both decrease with more observations (Table 15; Appendix D.7.1).

Table 15: Different errors measured by IQD.

Context sizeFinite-data errorApprox error
100.009 ± 0.0020.012 ± 0.004
200.007 ± 0.0010.010 ± 0.002
1000.003 ± 0.0010.008 ± 0.002

W4 & Q3: Could the authors provide more pretraining details and sensitivity analysis across independent pretraining runs?

A: Yes. We have added detailed descriptions of the pretraining and checkpoint selection to Appendix B.2 , along with results from 3 data generation seeds.

AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Thank you for your constructive comments. We have added experiments on prior sensitivity, unequal posterior weights, and finite-data versus approximation error, and clarified the theoretical assumptions and pretraining details. These updates are summarized in our rebuttal and detailed in the revised manuscript. We would appreciate your thoughts on these updates and would be happy to address any remaining questions.

AuthorsOpenReview ↗

Follow-up Before the Discussion Closes

Thank you again for your review. With the discussion period coming to an end tomorrow, we would greatly appreciate your feedback on whether our rebuttal has addressed your concerns. Your perspective is important to us, and we would be happy to clarify any remaining points before the discussion closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Contributions

We again thank the reviewers for their constructive feedback, which has helped us strengthen the empirical evaluation and clarify our work’s contributions and relationship to prior research. We have substantially revised the manuscript accordingly. Specifically, we have:

  1. Extended nonlinear evaluation from point predictions to full counterfactual distributions using a benchmark with an exact ground-truth posterior (Appendix D.9).
  2. Distinguished finite-data error from CAFE’s approximation error (Appendix D.7.1).
  3. Evaluated sensitivity to the pretraining prior, testing recovery of unequal posterior weights and robustness to changes in graph and noise priors (Appendices D.7.1–D.7.3).
  4. Evaluated predictive uncertainty beyond the ANM assumptions using CRPS and coverage metrics (Section 4.4).
  5. Added comparisons with CausalFM and Do-PFN (L2L_2 models) and TabPFN (L1L_1 model), adapted to individual counterfactual prediction (L3L_3 task) via CATE adjustment (Section 4.4).
  6. Clarified the pretraining procedure, including task generation, training configuration, and checkpoint selection (Appendices A.1 and B.2).
  7. Evaluated sensitivity to output-discretization (Appendix D.10).
  8. Expanded the discussion of related work, clarifying connections to counterfactual identification, causal foundation models, and established PFN results (Sections 1–3; Appendices C.1–C.2).

The remaining disagreement with Reviewer AEKi concerns methodological novelty. We acknowledge that CAFE builds on the PFN paradigm, but respectfully disagree that its contribution is adequately described as simply “conditioning on Y and restricting the prior.” We formalize the core contributions of CAFE as follows.

  • CAFE develops and validates a framework for individual counterfactual prediction. Counterfactual prediction is itself a challenging task in causal inference, which requires individual-specific information that interventional predictions generally do not determine. Following the research paradigm of causal PFNs (e.g., do-PFN and CausalFM), we formulate this task as amortized Bayesian posterior predictive inference and develop the corresponding input representation, model encoder and training objective. We further characterize the counterfactual posterior analytically and evaluate the framework through extensive experiments. We also evaluate TabPFN and causal PFNs, showing that their strong predictive or interventional capabilities do not translate into comparable counterfactual accuracy (Section 4.4). The challenges of individual counterfactual prediction, the distinct task formulation, and our empirical results together highlight the need for a model specifically designed and trained for this task.
  • CAFE rigorously characterizes the sources of counterfactual uncertainty. Although causal PFNs already amortize Bayesian inference over SCM priors and account for graph uncertainty, CAFE extends this line of work by isolating uncertainty due to graph ambiguity from finite-data uncertainty about causal mechanisms. Establishing this separation is not straightforward. It requires showing that, under ANM class and our stated regularity and support conditions, each compatible graph determines a unique counterfactual outcome when the observational distribution is known exactly. Any remaining uncertainty therefore comes from disagreement among compatible graphs, and CAFE captures this uncertainty in its amortized posterior predictions.
  • CAFE empirically demonstrates the value of pre-trained posterior predictive modeling for counterfactual prediction. Our experiments show that CAFE estimates uncertainty about individual counterfactual outcomes more accurately than DECI and DiBS, which are two Bayesian causal structure learning methods that explicitly approximate the posterior over causal graphs for each dataset using variational inference and a particle-based approximation, respectively (Appendix D.3). This comparison implies the effectiveness of pre-training-based posterior predictive modeling. Our experiments also show that CAFE makes precise predictions when counterfactuals are identifiable and preserves uncertainty when compatible graphs imply different outcomes (Sections 4.1–4.2). Its performance in realistic simulations also supports predictive robustness beyond its training assumptions (Section 4.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.