acceptodds
Under review as a conference paper at ICLR 2027

Amortized Counterfactual Prediction with Graph Neural Networks

Abstract

Counterfactual inference asks how an observed outcome might have differed under different circumstances, such as whether a patient would have recovered had they received a different treatment. Such questions arise whenever decisions depend on the outcomes of alternatives that were not observed. Answering counterfactual queries generally requires knowledge of the underlying causal structure, which is often unavailable in practice. A common approach is therefore to first estimate the causal graph from observational data and then perform counterfactual inference. However, errors in the estimated causal graph can propagate to downstream counterfactual predictions. To address this challenge, we introduce the , an end-to-end approach that answers counterfactual queries directly from observational data without first estimating a causal graph. CFGNP uses Graph Neural Networks (GNNs) to exploit the dual-graph structure by restricting cross-world message passing to corresponding factual–counterfactual variables. Across synthetic and image-based benchmarks, CFGNP achieves stronger predictive performance than existing approaches, including on the German Loan benchmark, where it reduces MSE from to . Furthermore, on synthetic causal graphs, a model trained only on 10-node graphs maintains strong predictive performance on graphs with up to 30 nodes, without retraining.

Primary area:
causal reasoning
Keywords:
Counterfactual Inference, Unknown Causal Structure

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6qgfOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses backtracking counterfactual prediction when the causal graph is unknown. In the absence of a causal graph, the usual pipeline estimates one from observational data and then performs counterfactual inference. This has two shortcomings: 1) the search space of causal graphs is super-exponential, and 2) graph estimation errors amplify downstream errors and incorrect counterfactual identification. The authors instead propose CFGNP, which predicts the counterfactual distribution directly from observational samples, without committing to a causal graph.

Critical strengths

S1. Crisp problem formulation. The research gap is genuine and well framed.

S2. One model across varying graphs. The baselines need a fixed causal graph, while CFGNP can handle multiple SCMs per sample.

S3. Strong synthetic results. Many Table 1 rows improve by orders of magnitude (for example, Triangle LIN 3.3×10−33.3 \times 10^{-3} down to 2.6×10−72.6 \times 10^{-7}).

S4. Architecture ablation. Appendix D.4 provides a strong justification for using simple graph neural networks over more complex models. Table 6 shows that replacing the Graph Attention Network (GAT v2) with a plain Transformer yields substantially worse results (for example, Triangle LIN: GAT's 2.59×10−72.59 \times 10^{−7} against Transformer's 8.33×10−48.33 \times 10^{−4})

S5. Readability. The paper is well written and easy to follow.

Critical weaknesses

W1. Untested claim. The introduction motivates CFGNP by arguing that the usual pipeline (estimate the graph, then run counterfactual inference) allows graph errors to propagate, whereas skipping graph estimation avoids this. This claim is not tested in the paper. The authors should run a causal discovery method (e.g., PC, NOTEARS, or DAG-GNN, as cited in their Related Work) on the same observational context, feed the estimated graph to the graph-based baselines (VACA, BGM, DeepBC), and then compare against CFGNP. This comparison aligns with the practical setting that the paper's introduction targets. Another related experiment should compare CFGNP against itself on a given causal graph. For example, restrict message passing to the true or estimated edges rather than the full graph, to show what the missing graph costs or gains.

W2. CheXpert lacks ground-truth counterfactuals. The evaluation uses classifier-measured attribute changes and qualitative image inspection. That is standard for image counterfactuals, but it does not validate counterfactual correctness; a semi-synthetic image benchmark with known generative factors would materially strengthen the real-data claim. Results on the CANDLE benchmark (Reddy & Balasubramanian, 2022) should settle this. It will allow the authors to measure whether the generated image changes the intended factor while preserving the others. The image baselines (VCI, DeepBC) should also be run on it for comparison.

Reddy, Abbavaram Gowtham, and Vineeth N. Balasubramanian. "On causally disentangled representations." AAAI 2022.

W3. VCI on the synthetic benchmarks. VCI, like the proposed method, does not require an explicit causal graph. Restricting it to CheXpert makes the headline synthetic dataset comparison asymmetric as CFGNP is compared with graph-known methods there, while its nearest graph-free competitor appears only in the image experiment.

W4. Context-size and sample-efficiency ablation in the main evaluation. CFGNP’s premise is that observational context OO identifies or constrains the SCM. The appendix examines convergence with context size, but the main experiments should report performance as a function of NobsN_\text{obs}, especially for small contexts where the model-averaged posterior should be highly uncertain.

W5. Uncertainty. The paper lists uncertainty estimation as a contribution, stating:

line 317: We use a mixture of Nc Gaussians, allowing CFGNP to represent uncertainty arising from ambiguity over the underlying SCM.

However, this claim isn't substantiated. The reported metrics are limited to MSE, ROC-AUC and accuracy; no uncertainty metrics (e.g. calibration error or negative log-likelihood) are reported, and there is no experiment demonstrating that the predicted uncertainty is meaningful or useful. The authors should evaluate uncertainty directly. They could also show how predictive uncertainty increases when the underlying SCM is less identifiable from the observed data, or its decrease with increasing observed data.

W6. OOD experiment.

  1. Comparing Tables 2 and 3, the OOD models beat the full model on multiple occasions, even the ones that never saw those features. This is counterintuitive.

  2. If a model never saw feature kk during training, it is expected to be worst at handling feature kk. However, most of the differences within a row are small compared with the ±\pm std values. What dominates the table is how hard each feature is in general (AP/PA easy, age and disease hard), not which feature was left out. That effect is not evident through this experiment.

  3. The columns in Table 2 indicate which variable wasn't seen during training. However, it is unclear what the row indicates. Does the row correspond to the variable being perturbed to create the counterfactual? For example, in the cell corresponding to row "Sex" and column "OOD Age", was the patient's sex perturbed to create the counterfactual, or could a different variable be perturbed, and are we just studying the counterfactual's consistency with respect to sex? Or are the row entries averaged across all possible targets?

  4. For the results in Table 2, the classifiers score the generated images. The image generator was trained with a reconstruction loss that keeps the counterfactual image close to the real one (equation 2). So whenever a feature is not the one being changed, the model can score well on it by copying the original X-ray. A row averaged over all queries therefore mostly measures "did the image stay close to the original", not "did the model understand this feature". Addition of a "return the original image unchanged" baseline, to show how much of each score comes from copying would help.

W7. Stale references. I noticed that some of the published references have arXiv citations. For example:

  • VCI has been published at ICLR 25.
  • Ribeiro et al. is published at NeurIPS 25.

Please audit all the references.


Note

I have set my review confidence to 2. I specialise in graph neural networks, but I'm not an expert in causality. I am familiar with causality, but I don't follow its research closely.

Questions for authors

Q1. OOD training. Line 451 writes, "we exclude one feature k_OOD during training". How exactly is this done? Are nodes corresponding to that feature deleted from the input graphs during training? Or is the loss function modified to ignore that variable. Please clarify.

Q2. OOD experiment. What is the target variable in each of table 3's rows? Was the the feature column also the target?

Q3. Conversion to point estimates. Please elaborate the exact procedure of reducing the mixture outputs to the point estimates reported in the tables.

4 comments

We thank the reviewer for their comments. When incorporating the comments, we reran old experiments as well to ensure that all methods were trained on exactly the same dataset. Since the construction is stochastic, results have changed in many cases. Based on the previous std of the loss, it seems that some seeds had many outliers. This is not the case anymore.

W1: Untested Claim. We thank the reviewer for their comment. We use NOTEARS to get the causal graph and subsequently ran the baselines on the sampled causal graphs. Results have gotten considerably worse, for instance BGM from 4.52 × 10^−4 MSE to 4.32 × 10^−3 on triangle_LIN. The exception is triangle_NADD where results got better, even than CFGNP. Full results are in App. E.10 (p.42-43), referred to on l.364.

W2: Chexpert lacks ground-truth counterfactuals. We thank the reviewer for this suggestion and have added experiments on CANDLE. For each test image, we intervene on each of the six factors in turn, setting it to a new random value. A ResNet-18 factor classifier (0.82 test accuracy) reads the factors from the generated images, and we report three metrics: Effectiveness: the intervened factor takes its new value. Preservation: all other factors stay the same. Composition: the reconstruction MSE under a null intervention.

We also run VCI and DeepBC on the same split, using a shared encoder/decoder. App. E.5 (p.39, referred to in l.349) shows the results. CFGNP trades effectiveness for preservation, performing as well as DeepBC (lower effectiveness but much higher preservation). VCI performs worse.

W3: VCI on synthetic benchmarks VCI is now included in the synthetic experiments (Table 1, p.8). VCI performs considerably worse than the other baselines and CFGNP. For instance, for german loan its MSE is only 1.95 × 10^−1.

W4: Context-size and sample-efficiency ablation in the main evaluation. We used a CFGNP pretrained on 100 observations for inference on smaller context sizes {1, 5, 10, 25, 50, 75, 100}. Results get worse on smaller context sizes (App. E.7, p.41).

W5: Uncertainty. We thank the reviewer for their suggestion. We added a synthetic experiment evaluating CFGNP on increasing observational context sizes, measuring the predictive std and the KL divergence. The predictive std is close to the analytically computed Bayes-optimum. Both go down with increasing context size and approach the predictive std of the true counterfactual mechanism. The KL divergence between CFGNP and the Bayes-optimal decreases slightly with increasing context size, possibly because the model has more issues dealing with noise for small sample sizes. More importantly, the KL divergence between the true counterfactual distribution and CFGNP is much bigger for small context sizes and approaches the other KL divergence for larger contexts. This shows that CFGNP does quantify the increased uncertainty for smaller observational context, i.e. when the causal graph is less identifiable from observational data. Details are in App. E.6 (p.40) and in the main results section (paragraph “Predictive Uncertainty”, p.9, l.482-).

W6: OOD experiment. We thank the reviewer for scrutinising the OOD experiment. On closer inspection, the results do not support our claim that CFGNP generalizes to variables unseen during training, as the models seem to have learned to just copy the input. We have therefore removed this claim from the abstract, introduction and conclusion, and report the experiment in App. E.1.2 (p.34) as a limitation. The issue is specific to this setting and does not affect the other results. In Table 4 (p.34) we have added a baseline that just predicts the input. After splitting the features into changed features (the intervention index) and preserved features (the rest), the scores of the baseline and the OOD models are very similar, except for the age feature.

W7: Stale references. We have updated the references to the published versions.

Q1: OOD Training We now state in the document that the excluded feature k_OOD is removed by deleting the corresponding node from the graph during training (p.34, l. 1813).

Q2: OOD Experiment We have updated the table to only show changed features and preserved features (now Table 4, p.34).

Q3: Conversion to point estimates During training we use all outputs of the model to additionally obtain the variance, which we can use to quantify uncertainty. For the reported scores during evaluation, we use the weighted sum of μ and w (App. B.5, p.25, l. 1325).

Thank you for the revision. Most of my concerns have been addressed, but a few remain, as detailed below.

W1b Untested claim

Review: Another related experiment should compare CFGNP against itself on a given causal graph. For example, restrict message passing to the true or estimated edges rather than the full graph, to show what the missing graph costs or gains.

Unresolved. The authors did not conduct this experiment.

W2. CANDLE

Partially resolved.

  1. Preservation can be gamed by copying. A model that returns the input unchanged would score close to the oracle ceiling on preservation and close to zero on effectiveness. The authors mention that their method "trades effectiveness for preservation". However, copying does the same. Hence, this experiment needs a copy baseline to put CFGPN's numbers in context. Please add the copy baseline to Table 7, as already done for CheXpert in Table 4.
  2. Line 349 of the current revision writes, "CFGNP trades effectiveness for preservation, performing as well as the baselines." I'd revise it, as DeepBC wins on both effectiveness and composition.

Note.

W2. It will allow the authors to measure whether the generated image changes the intended factor while preserving the others.

In hindsight, my request for a preservation metric assumes interventional semantics while the paper uses backtracking counterfactuals. Interventional counterfactuals hold upstream variables fixed while backtracking counterfactuals do not. So a backtracking counterfactual making an object large could legitimately change an indoor scene to an outdoor one (a non-target attribute change), since the confounder links size and scene. I'd welcome a brief comment on whether full preservation is the right target for backtracking counterfactuals.

Connected to this line of thought, the authors mention in App. E.5:

In the causal graph of CANDLE the factors do not cause each other (they are related only through confounders) and all of them cause the image. Changing one factor should therefore change exactly that factor in the image and leave all other factors unchanged, which makes the correctness of a counterfactual image measurable.

Since CFGNP predicts backtracking counterfactuals, under which confounders may change and legitimately alter non-target factors, preservation of all non-target factors may not the right target.

W5. Uncertainty.

The reported metrics are limited to MSE, ROC-AUC and accuracy; no uncertainty metrics (e.g. calibration error or negative log-likelihood) are reported, and there is no experiment demonstrating that the predicted uncertainty is meaningful or useful. The authors should evaluate uncertainty directly.

Unresolved: The authors have not reported the uncertainty estimates of the datasets used in the experiments.

They could also show how predictive uncertainty increases when the underlying SCM is less identifiable from the observed data, or its decrease with increasing observed data.

Partially resolved: The authors have somewhat addressed the last part (decreasing uncertainty with increasing observed data). However, they have tested it on a two-node SCM, which is too simplistic. They should present the uncertainty change on the actual datasets used in the paper.


All remaining weaknesses are resolved. W6 (OOD experiments): resolved by the authors retracting the claim and removing it from the paper.

PS. Interpreting Table 10 on page 43 requires comparing numbers from Table 1 on page 8. For the reader's convenience, add a copy of Table 1 alongside Table 10 in the appendix, in a similar format.

W1b.

We thank the reviewer for the comment and apologize for the confusion. We had already included it in the rebuttal (Table 2, p.9) but forgot to mention that in the reply to the comment. Restricting Message-Passing to the true causal graph (with observations) reduced performance across 6 out of 7 datasets compared to normal CFGNP. In one it even performed worse than CFGNP without context. However, based on reviewer Nd2t’s comment, we have now supplied the ground-truth to CFGNP via edge labels. This keeps Proposition A.1 and improves performance. Now, CFGNP with observational context and true causal graph for instance performs best on German loan, Mshape_LIN and Mshape_NADD and is within on std of the best results on the other datasets. CFGNP without observational context but with true graph now performs best on Triangle_LIN and Triangle_NADD. Vanilla CFGNP performs best in the rest. We have updated Table 2 (p.9) accordingly. We have also included an ablation between the previous and the new label-based approach in section E.12 (p.44).

W2

We thank the reviewer for their comment. We have added a copy baseline, both one that just copies the image and one that copies the hidden state inside a VAE (because this is what CFGNP uses). For all values, see Table 8 (Sec. E.5, p.40). The VAE-copy has considerably lower effectiveness with 0.069 compared to CFGNPs 0.316. Preservation is slightly higher (0.765 vs. 0.694) and composition MSE is almost 0 (0.0010) as expected. We have updated the manuscript to: “CFGNP has the highest preservation; DeepBC has higher effectiveness and better composition.” (p.7, l.348)

Note Reply.

We thank the reviewer for this point. We agree that under backtracking semantics, non-target factors that share a confounder with the intervened factor may legitimately change. In CANDLE, however, this dependence is limited. We measured it for all pairs of factor labels using Cramér’s V with a permutation test (Table 9, App. E.5, p.40). Only four of the 15 pairs are dependent, all with moderate strength (largest bias-corrected V=0.36), and rotation and light are independent of all other factors. Since a backtracking counterfactual can change a non-target factor only through a shared confounder, interventional and backtracking counterfactuals agree on the non-target factors for most interventions, and preservation measures the same quantity under both semantics. Moreover, on CANDLE we train CFGNP with the interventional target (intervened factor changed, all other factors unchanged), so preservation is the objective the model is trained for.

W5.

We thank the reviewer for their comment. We have added Table 10 (App. E.8, p.42) that reports the calibration error as the uncertainty metric. We compare to all baselines where we use counterfactual supervision with an NLL loss (Details in App. D, p.32). Additionally we compare to vanilla VACA and VCI as they also produce probability distributions. CFGNP has the lowest calibration error on German loan. Otherwise, especially vanilla CFGNP has a lower error. VCI sometimes has a lower calibration error. The other two baselines with NLL supervision have considerably higher calibration error. As both VCI and VACA have the purpose of producing probability distributions, they are also developed to have low calibration errors.

However, they have tested it on a two-node SCM, which is too simplistic.

We chose the two-node SCM because it is a setting in which the true counterfactual distribution given the observed context can be computed analytically. This allows us to verify that the predicted uncertainty matches the correct uncertainty, not only that it decreases. As requested, we now also report the change in uncertainty on all seven benchmark datasets (Fig. 25, p.43). We train CFGNP on 100 observations and then evaluate it only on an observational context form 1 to 100. We report the predicted std, the 90% interval coverage and the NLL. On all datasets, the predicted standard deviation decreases monotonically with context size and levels off at a dataset-dependent floor, highest for the NADD case. At the training context size, coverage is close to nominal (0.87–0.97). Since the models are trained with 100 context samples, they are overconfident for very small contexts, which lie outside the training regime.

PS. Interpreting Table 10 on page 43 requires comparing numbers from Table 1 on page 8. For the reader's convenience, add a copy of Table 1 alongside Table 10 in the appendix, in a similar format.

We have attached the copy of Table 1 alongside Table 10.

Thank you for the response. It addresses my remaining concerns. I am raising my rating from weak acceptance (3) to clear acceptance (4).

Reviewer_iNrzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the Counterfactual Graph Neural Process (CFGNP), a method for answering counterfactual queries without first estimating a causal graph. A Neural Process conditions on a set of observational samples, and a graph neural network passes messages over a dual graph that restricts cross-world edges to corresponding factual and counterfactual variables, so the model maps observations and a query directly to a counterfactual prediction. The method is evaluated against counterfactual inference methods that are given the causal graph or its topological ordering, on three (semi-)synthetic structural causal models, on randomly generated causal graphs of 10 to 30 nodes, and on CheXpert chest X-rays.

Critical strengths
  1. The counter-factual inference without causal graph is interesting, and the method is principled.

  2. The experiments look seriously conducted and well analyzed.

Critical weaknesses
  1. CFGNP is trained on ground-truth counterfactual pairs and no baseline is, so the comparison is unfair. BGM and VACA are given the true causal graph and CFM its topological ordering, but none is given counterfactual labels, while CFGNP is (p5, p7). No ablation removes the advantage, no baseline is given a supervised head, and there is no trivial control.

  2. The quantity being estimated is never specified, and the one evaluation with real images runs through its own training signal. The paper commits to backtracking counterfactuals (p4), which are defined only relative to a cross-world coupling p(U,U∗)p(U,U^*) that is never specified for any benchmark; the only coupling written anywhere is the minimum-change one of Appendix C.5, under which the paper itself notes that backtracking and interventionist counterfactuals coincide. So a reader cannot tell what Tables 1 and 6 are errors against. On CheXpert, the attribute classifiers used for evaluation also appear in the training objective (Eq. 3, p24), and the reported counterfactual age MAE of 2.71 is less than half the measuring regressor's own MAE of 6.1473.

  3. The framing is the published recipe for amortized interventional inference with one extra query input, and the two closest prior works are uncited.

Do-PFN (Robertson et al., 2025) and CausalPFN (Balazadeh et al, 2025) both posit a prior over SCMs, pre-train on simulated data and answer in one forward pass without the causal graph, at the population-interventional level; Do-PFN's Appendix D.1 already holds the noise constant across the intervention and its limitations name counterfactual queries as future work.

  1. The code is not provided, so reproducibility is not possible.
Questions for authors

To improve the paper the most, consider:

  1. Quantify the supervision advantage (p5, p7). Please train CFGNP without counterfactual pairs, or give one baseline a supervised head with the true graph, and add a trivial control that predicts the interventional or conditional mean. As it stands the margins cannot be attributed to avoiding graph estimation rather than to the labels, and this single experiment would settle it.

  2. Do Table 1's train and test SCMs coincide? Section 5's contrast with the graph-size setting, "where each sample comes from its own SCM" (p7), implies they do. Please state it either way and report dataset and split sizes, which are currently given only for the two-node toy (p35). If they coincide, Table 1 is supervised regression on one fixed function rather than evidence of meta-learning.

  3. Which coupling p(U,U∗)p(U,U^*) generated the benchmark targets? If it is the deterministic minimum-change coupling, backtracking and interventionist counterfactuals coincide then the dual graph needs a different justification.

3 comments

We thank the reviewer for their comments. We note in the beginning that, when incorporating the reviewers comments, we reran old experiments as well to ensure that all methods were trained on exactly the same dataset. Since the construction is stochastic, results have changed in many cases. Based on the previous std of the loss, it seems that in some cases there where seeds with many outliers. This is not the case anymore.

Q1: Supervision advantage. We thank the reviewer and ran the following controls: Supervised baselines. We have retrained each baseline in two different configurations. One in the proposed way without counterfactual supervision, and the second configuration where we have added a counterfactual supervision loss. Still all baselines receive the correct graph. The counterfactual supervision improves the loss, while it still is worse than CFGNP without causal graph, showing its benefit. In particular for the loan dataset, baselines with supervision get close to the results of CFGNP.. For instance BGM on German loan with MSE of 2.11 × 10^−4 compared to CFGNP with 6.57× 10^−5. The results can be found in table 1 on page 8. CFGNP ablations. These can be found in Table 2 on p.9. In one experiment, CFGNP receives no observations and no causal graph. This performs much worse than CFGNP, for instance on Triangle_NADD with MSE of 1.7 × 10^-4 compared to 4.08× 10^−5 in normal CFGNP. This shows that counterfactual supervision alone is not sufficient. In a second experiment we provide the dual-graph architecture with the true causal graph. This performs better than without the causal graph but worse than CFGNP (MSE on loan of 7.60 × 10^−5). This may be the case because CFGNP obtains information about both the causal graph and structural equations from the observations. Finally, we provide CFGNP with observations and causal graph, which only improves performance on M-shape_NADD (MSE 1.19 × 10^−4). In this case the proof of the theoretical properties of CFGNP does not hold anymore which may explain the result. (Proposition A.1, page 17). The interventional mean E[X | do(X_j = α)] and the conditional mean E[X | X_j = α], computed exactly under the true SCM, ignore the factual unit. CFGNP’s error is about 10³× lower on Triangle and M-shape and 10^4× lower on Loan, so it predicts individual counterfactuals rather than average effects. The results can be found in appendix E.9 (p.42) in Table 9.

Q2: Causal graphs in train and test set. We thank the reviewer for their feedback. Table1’s train and test SCMs do in fact coincide. We are aware that this is not technically meta-learning, however we still think the experiments are helpful to compare CFGNP to state-of-the-art methods. We have added the dataset and split size in the appendix (App. C.3, p.27, l.1409). For all experiments use in Table 1 (p.8) it is the following: They are split into 26250 train samples, 11250 validation samples, and 12500 test samples.

Q3: Cross-world coupling. We thank the reviewer for their comment. For the experiments in Table 1 we copy the counterfactual generation mechanism exactly from [1] to allow for better comparability. Here, the samples are interventional counterfactuals (now mentioned in App. C.3, p.27, l.1413). However, in the other experiments, in particular the size experiments in Figure 4, the coupling between factual and counterfactual world is taken as p(U*|U)=N(U,2) (now explained in App. C4, p.29, l.1564), exhibiting the behaviour of backtracking counterfactuals. We use backtracking counterfactual in our experiments because they do not violate the laws of nature [2]. This is also why we argue our dual graph is appropriate for predicting counterfactuals. Because even if in rare cases in reality, interventional counterfactuals did arise, one might not know that in advance and cannot adjust the architecture. Therefore, we choose the architecture that should be accurate in most normal cases.

[1] Pablo Sanchez-Martin, Miriam Rateike, and Isabel Valera. VACA: Designing Variational Graph Autoencoders for Causal Queries. In Proceedings of the AAAI Conference on Artificial Intelligence [2] Julius Von Kügelgen, Abdirisak Mohamed, Sander Beckers Proceedings of the Second Conference on Causal Learning and Reasoning, PMLR 213:177-196, 2023.

W3: Missing citation. We thank and acknowledge the reviewers concerns for the missing sources. We have now included these sources in related work. It now contains the reference “Do-PFN (Robertson et al., 2025) and CausalPFN (Balazadeh et al., 2025) pre-train transformers…” (l. 135)

W4: Missing code. We thank the reviewer for their comment on the missing code. We have now uploaded the anonymized code at https://anonymous.4open.science/r/CFNP-2150/. We have included this in the paper as well (l.560).

I am raising my score to 3 as some of my concerns have been addressed (W1 mostly, W2 partly, the CheXpert concern remains, W3, W4), and appreciate the responses to Q1-Q3.

  • W1 follow-up. The response states that all methods were retrained because they had previously been trained on different datasets, and that results changed in many cases. How far did the headline numbers move?

  • W2 follow-up. If the dual graph is justified because backtracking does not violate the laws of nature, why is the headline evaluation interventional? Table 1 reuses VACA's generation code, which keeps the counterfactual noise equal to the factual noise, while only the Figure 4 size experiments use N(U,2)\mathcal{N}(U,2). And why variance 2, given that the coupling defines the target rather than being a tunable parameter?

  • New, from the Q1 ablations. Giving the architecture the true causal graph makes it worse, 7.60 × 10⁻⁵ against 6.57 × 10⁻⁵ on Loan. What explains a model degrading when handed the ground truth? .

We thank the reviewer for their positive feedback.

W1.

The results, mentioned for German loan changed from 1.9 × 10−3 to 6.57 × 10^−5 for CFGNP. The baselines improved from 4.51 × 10^−2 to 2.11 × 10^−4. There was one outlier dataset beforehand. Results across datasets are now 1.6–7× better compared to baselines.

W2.

We thank the reviewer for their comment. We chose the benchmarks from the VACA paper because they are regularly used in related work. Interventional counterfactuals are often used because Pearl’s abduction–action–prediction originally made them the standard definition. Changing the counterfactuals of these datasets from VACA could confuse readers who are familiar with them. We therefore kept them interventional, which also shows that CFGNP handles this case. Backtracking counterfactuals are our main motivation because they keep every mechanism intact. This is why we use them in Figure 4. Regarding the coupling in Figure 4, we apologize for the confusion. There apparently was an issue when pasting the responses. The actual coupling is N(U_i, \sigma_i^2). We agree the coupling defines the target. That is why we don’t fix it: drawing \sigma_i per task and node makes Fig. 4 average over a range of closeness levels, and the model never observes \sigma_i. We have adjusted this nodewise sampling in the manuscript.

W3.

We thank the reviewer for their comment. This is likely because the restricted graph hinders message passing in the GNN, especially between nodes with further graph distance. On the other hand, the full graph facilitates easy message passing and keeps the Proposition A.1. Based on reviewer Nd2t’s comment, we have now supplied the ground-truth to CFGNP via edge labels. This keeps Proposition A.1 and improves performance. Now, CFGNP with observational context and true causal graph for instance performs best on German loan. We have also included an ablation between the previous and the new label-based approach in section E.12 (p. 44).

Reviewer_Nd2tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Counterfactual Graph Neural Process (CFGNP), an end-to-end approach for counterfactual prediction that does not require an explicitly specified causal graph. Instead, it conditions predictions on observational samples, a factual instance, and a counterfactual query, using graph neural networks to exploit the structure connecting factual and counterfactual variables. The main motivation is to avoid the difficulty and potential error propagation associated with estimating a causal graph before performing counterfactual inference. The authors report improved results on synthetic benchmarks and CheXpert, as well as generalization to larger graphs and unseen features.

Critical strengths

None

Critical weaknesses
  1. The paper reports improvements on some benchmarks, but it is not sufficiently clear whether these gains are practically meaningful compared with existing methods. If established baselines already achieve low prediction errors (on the order of 10−210^{-2} in MSE), the authors should explain what additional benefit CFGNP provides and whether its improvements matter for downstream counterfactual decisions.

  2. The paper would benefit from a clearer articulation of its main contribution. Is the primary advantage of CFGNP its ability to perform counterfactual prediction without a prespecified causal graph, or its predictive performance? The evaluation does not fully isolate the benefit of relaxing this assumption from the benefit of using a different model architecture and training procedure.

  3. CFGNP uses counterfactual pairs during training, unlike the compared baselines (unless I'm missing something). This raises questions about the fairness of the comparison and the method's practical applicability, since counterfactual outcomes are generally not directly observed. The authors should clarify how these pairs are obtained and evaluate the method under comparable data-access assumptions.

  4. Although CFGNP performs well on several benchmarks, it does not consistently outperform all existing methods. The paper should provide a more nuanced discussion of the conditions under which CFGNP is advantageous, and investigate why existing approaches perform better in some settings.

Questions for authors

Q1: Is the main contribution of CFGNP its ability to perform counterfactual prediction without a known causal graph, or its improved predictive accuracy? Could the authors clarify and evaluate these contributions separately?

Q2: CFGNP uses counterfactual pairs during training, whereas the compared baseline methods do not. How can the authors ensure that the performance comparison is fair given this difference in training information?

Q3: CFGNP does not achieve the best MSE on every benchmark. Under what conditions does CFGNP outperform existing methods, and when should alternative methods be preferred?

Q4: Which components of CFGNP contribute most to its performance?

Q4.1 Could the authors isolate the effects of the dual-graph architecture, training objective, and counterfactual pairs through controlled ablation studies?

3 comments

We thank the reviewer for their comments. We note in the beginning that, when incorporating the reviewers comments, we reran old experiments as well to ensure that all methods were trained on exactly the same dataset. Since the construction is stochastic, results have changed in many cases. Based on the previous std of the loss, it seems that in some cases there where seeds with many outliers. This is not the case anymore.

W1: Practical significance. We thank the reviewer and agree with the core of this point. On Triangle and M-shape, BGM and CFM already reach MSEs of 10^-5 (the versions with counterfactual supervision). However, errors often carry on to downstream tasks [1]. Therefore, even small improvements can make a difference. [1] Karimi et al., NeurIPS 2020, Algorithmic recourse under imperfect causal knowledge: a probabilistic approach.

W2/Q1: Clarification of Contributions. We thank the review for their comment. While the contribution of CFGNP is the novelty of counterfactual inference without a causal graph, the improved predictive accuracy validates our approach and the usefulness. We have made the contributions more clear in the paper. The abstract now mentions “CFGNP both offers an approach that does not require knowledge of the causal graph and achieves stronger predictive performance than existing approaches.” (line 23-25) In the introduction we clarify why we do require good results: “We demonstrate the validity of the CFGNP…” (l 102-103)

W3/Q2: Fairness of Comparison. We recognize the issue that comparing CFGNP to the baselines that are not build to receive any counterfactual supervision. As this is a new approach to the problem, there exists no method that receives similar input. In particular, there exists no method with counterfactual supervision. However, we would like to keep a comparison to the baselines in order to make the possible gain in counterfactual inference performance clear. We have now included for each baseline, the original method as well as a modification where, during training, the loss is extended by a counterfactual supervision loss. “For each of the baselines we add an additional variant where we extend the train loss by a counterfactual supervision loss.” (l.402, p.8) The new results can be found in Table 1 (p.8). While results improve, for instance BGM on German loan with MSE of 2.11 × 10^−4, they are still worse than CFGNP. In the synthetic experiments the pairs are simulated from the SCM. In practice, CFGNP is meant to be used in an amortized way: train it on pairs simulated from a prior over SCMs, then apply it to a new dataset given only observational samples.

Q3: When to use CFGNP. We thank the reviewer for their comments. CFGNP’s advantage is largest when the causal graph varies across instances (Fig. 4) or when existing methods are far from exact on a fixed SCM, e.g. M-shape_NADD (from 1.37 × 10^−3 to 2.04 × 10^−4). Knowing the graph alone does not close this gap: the baselines in Table 1 receive the true graph, yet CFGNP is more accurate in 7 of 7 settings. When the graph is known and the mechanisms are simple (M-shape LIN), specialised methods are already near-exact and offer a cheaper alternative, as they train faster and require no counterfactual training pairs. We have added this discussion to Section 5 (l. 517-522, p.10), “When does CFGNP help”.

Q4: Ablation Studies. We thank the reviewer for their question. We have extended the ablation studies in the appendix by an ablation using the MSE loss. In order to isolate the effects of the dual graph architecture, we have replaced it with a simple transformer. This drastically reduces the performance, for instance the MSE for loan increases to 0.46. We further added an ablation where we use MSE instead of NLL. While this also lowers performance, we note that this is, from a theoretical perspective, not the correct loss since we aim to produce probability distributions. The above ablations are in Appendix E.8, p. 41-42.

Further, we have added ablations in order to quantify the effect of counterfactual supervision. These can be found in Table 2 on p.9. In one experiment, CFGNP receives no observations and no causal graph. This performs much worse than CFGNP, for instance on Triangle_NADD with MSE of 1.7 × 10^-4 compared to 4.08× 10^−5 in normal CFGNP. This shows that counterfactual supervision alone is not sufficient. In a second experiment we provide the dual-graph architecture with the true causal graph. This performs better than without the causal graph but worse than CFGNP (MSE on loan of 7.60 × 10^−5). This may be the case because CFGNP obtains information about both the causal graph and structural equations from the observations. Finally, we provide CFGNP with observations and causal graph, which only improves performance on M-shape_NADD (MSE 1.19 × 10^−4). In this case the proof of the theoretical properties of CFGNP does not hold anymore which may explain the result. (Proposition A.1, page 17).

I thank the authors for the extensive revision; the -CF baselines and new ablations are welcome and address part of my concerns. Several points remain:

Amortization is not tested in Table 1. Per App. C.3, train and test pairs for each benchmark come from the same fixed SCM, so the observational context carries no task-specific information and CFGNP effectively acts as a supervised regressor on oracle counterfactual pairs. The deployment scenario described in the rebuttal (train on a prior over SCMs, apply to a new dataset) is only exercised in Fig. 4, where baselines are fit to a single fullest DAG over heterogeneous SCMs and are therefore disadvantaged by design.

Could the authors (a) evaluate CFGNP trained on a prior over SCMs and tested on held-out SCMs (e.g., the Table 1 benchmarks unseen during training), and (b) compare in Fig. 4 against baselines fit per task on that task’s context (with true or estimated graph), stating whether these receive counterfactual supervision?

CF baselines. App. D adds an MSE counterfactual loss with λ=1 and untuned hyperparameters, while Table 8 shows MSE training degrades CFGNP by up to an order of magnitude.

Please tune λ/hyperparameters for the -CF baselines, or add an NLL-based supervision variant where applicable.

Table 2 interpretation. “w/o context, true graph” is within one std of CFGNP on Triangle NADD and German Loan, and adding context to the true-graph variant worsens results in 5/7 settings. Since restricting edges also breaks Prop. A.1, these ablations conflate graph knowledge with message-passing expressivity.

Could the authors add a variant that keeps full connectivity but encodes the true graph as edge features?

Table 10. With an empty estimated graph on NADD, VACA reaches ~1.95e-5 on Triangle NADD, better than both CFGNP and VACA with the true graph.

Please explain this, and add a “copy factual + set intervened value” control to E.9.

Please report old vs. new results for the rerun experiments, and reconcile the unseen-feature generalization claim (l. 346) with App. E.1.2, where the OOD models appear to revert to copying the input.

As it stands, the evidence still does not separate the benefit of relaxing the causal-graph assumption from the benefit of oracle counterfactual supervision on a fixed SCM, so I maintain my score for now.

A1. Amortization is not tested in Table 1:

We thank the reviewer for their comment. For (a) we have trained CFGNP on a prior of random causal graphs and then perform zero-shot prediction on the test datasets of German Loan, Triangle, and M-Shape. For the baselines, we have trained them individually on the observational context of each sample with the causal graph estimated by NOTEARS. We do this as every sample in the test set can be seen as an individual task. In these experiments CFGNP performs best on across all datasets. On German Loan it is close to the baselines (1.03 × 10^−2 vs. 1.06 × 10^−2) while on the other datasets the margin is bigger (1.33 × 10^−3 vs. 7.18 × 10^−3). Results and a description of the SCM prior can be found in App. F (p.45-46). We also refer to it in the main results section: “Zero-shot Transfer. …”(l. 428-)

For (b) ran the baselines fit per task with the causal graph estimated by NOTEARS. This improved the results a lot, especially for BGM. We have included the curves into the figures, in particular Fig. 4 (p.9). CFGNP still beats BGM in 25 of 35 of the context size–mechanism pairs.

A2. CF baselines:

We thank the reviewer for their comment. As requested, we have added an ablation where we do counterfactual supervision with an NLL loss objective. In Table 4 (App. D) we compare the vanilla baselines to MSE supervision as well as NLL supervision. Supervision with the NLL objective beats the MSE objective only in two datasets. Therefore, we only use the MSE objective when comparing the baselines with counterfactual supervision to CFGNP.

A3. Table 2 Interpretation: We thank the reviewer for their proposal. We have implemented the proposed approach to supply CFGNP with the ground-truth causal graph. We use 7 labels to cover every possible relation between nodes. App. E.12 contains an ablation about which of the two approaches to take with metrics for both in Table 14. The label-based approach works considerably better and beats the previous approach in 6 out of 7 datasets. Therefore, we have selected the label-based approach to compare to vanilla CFGNP. Now, CFGNP with observational context and the true causal graph performs best in 3 out of the 7 datasets. Furthermore, in all of the other datasets, it is within one standard deviation of the best model. We have adjusted Table 2 (p.9) and the manuscript.

A4. Table 10: We thank the reviewer for their comment. We have now corrected the losses of the NOTEARS methods. For instance, the score of VACA with (empty) estimated graph is now 7.81 × 10^−3 and does not exceed neither VACA with true graph nor CFGNP (Table 13, p.45). As requested, we have added the “copy + set intervened value” baseline to Table 12 in E.10 (prev. E.9). The copy baseline is performs better than the other trivial baselines in 2 out of 7 datasets. However, they are generally close. We have also made it more clear, that CFGNP does not generalize well to OOD features: “App. E.1.2 reports an experiment on unseen CheXpert features, where CFGNP does not perform well.” (p. 7, l.351).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.