- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper addresses backtracking counterfactual prediction when the causal graph is unknown. In the absence of a causal graph, the usual pipeline estimates one from observational data and then performs counterfactual inference. This has two shortcomings: 1) the search space of causal graphs is super-exponential, and 2) graph estimation errors amplify downstream errors and incorrect counterfactual identification. The authors instead propose CFGNP, which predicts the counterfactual distribution directly from observational samples, without committing to a causal graph.
Critical strengths
S1. Crisp problem formulation. The research gap is genuine and well framed.
S2. One model across varying graphs. The baselines need a fixed causal graph, while CFGNP can handle multiple SCMs per sample.
S3. Strong synthetic results. Many Table 1 rows improve by orders of magnitude (for example, Triangle LIN down to ).
S4. Architecture ablation. Appendix D.4 provides a strong justification for using simple graph neural networks over more complex models. Table 6 shows that replacing the Graph Attention Network (GAT v2) with a plain Transformer yields substantially worse results (for example, Triangle LIN: GAT's against Transformer's )
S5. Readability. The paper is well written and easy to follow.
Critical weaknesses
W1. Untested claim. The introduction motivates CFGNP by arguing that the usual pipeline (estimate the graph, then run counterfactual inference) allows graph errors to propagate, whereas skipping graph estimation avoids this. This claim is not tested in the paper. The authors should run a causal discovery method (e.g., PC, NOTEARS, or DAG-GNN, as cited in their Related Work) on the same observational context, feed the estimated graph to the graph-based baselines (VACA, BGM, DeepBC), and then compare against CFGNP. This comparison aligns with the practical setting that the paper's introduction targets. Another related experiment should compare CFGNP against itself on a given causal graph. For example, restrict message passing to the true or estimated edges rather than the full graph, to show what the missing graph costs or gains.
W2. CheXpert lacks ground-truth counterfactuals. The evaluation uses classifier-measured attribute changes and qualitative image inspection. That is standard for image counterfactuals, but it does not validate counterfactual correctness; a semi-synthetic image benchmark with known generative factors would materially strengthen the real-data claim. Results on the CANDLE benchmark (Reddy & Balasubramanian, 2022) should settle this. It will allow the authors to measure whether the generated image changes the intended factor while preserving the others. The image baselines (VCI, DeepBC) should also be run on it for comparison.
Reddy, Abbavaram Gowtham, and Vineeth N. Balasubramanian. "On causally disentangled representations." AAAI 2022.
W3. VCI on the synthetic benchmarks. VCI, like the proposed method, does not require an explicit causal graph. Restricting it to CheXpert makes the headline synthetic dataset comparison asymmetric as CFGNP is compared with graph-known methods there, while its nearest graph-free competitor appears only in the image experiment.
W4. Context-size and sample-efficiency ablation in the main evaluation. CFGNP’s premise is that observational context identifies or constrains the SCM. The appendix examines convergence with context size, but the main experiments should report performance as a function of , especially for small contexts where the model-averaged posterior should be highly uncertain.
W5. Uncertainty. The paper lists uncertainty estimation as a contribution, stating:
line 317: We use a mixture of Nc Gaussians, allowing CFGNP to represent uncertainty arising from ambiguity over the underlying SCM.
However, this claim isn't substantiated. The reported metrics are limited to MSE, ROC-AUC and accuracy; no uncertainty metrics (e.g. calibration error or negative log-likelihood) are reported, and there is no experiment demonstrating that the predicted uncertainty is meaningful or useful. The authors should evaluate uncertainty directly. They could also show how predictive uncertainty increases when the underlying SCM is less identifiable from the observed data, or its decrease with increasing observed data.
W6. OOD experiment.
-
Comparing Tables 2 and 3, the OOD models beat the full model on multiple occasions, even the ones that never saw those features. This is counterintuitive.
-
If a model never saw feature during training, it is expected to be worst at handling feature . However, most of the differences within a row are small compared with the std values. What dominates the table is how hard each feature is in general (AP/PA easy, age and disease hard), not which feature was left out. That effect is not evident through this experiment.
-
The columns in Table 2 indicate which variable wasn't seen during training. However, it is unclear what the row indicates. Does the row correspond to the variable being perturbed to create the counterfactual? For example, in the cell corresponding to row "Sex" and column "OOD Age", was the patient's sex perturbed to create the counterfactual, or could a different variable be perturbed, and are we just studying the counterfactual's consistency with respect to sex? Or are the row entries averaged across all possible targets?
-
For the results in Table 2, the classifiers score the generated images. The image generator was trained with a reconstruction loss that keeps the counterfactual image close to the real one (equation 2). So whenever a feature is not the one being changed, the model can score well on it by copying the original X-ray. A row averaged over all queries therefore mostly measures "did the image stay close to the original", not "did the model understand this feature". Addition of a "return the original image unchanged" baseline, to show how much of each score comes from copying would help.
W7. Stale references. I noticed that some of the published references have arXiv citations. For example:
- VCI has been published at ICLR 25.
- Ribeiro et al. is published at NeurIPS 25.
Please audit all the references.
Note
I have set my review confidence to 2. I specialise in graph neural networks, but I'm not an expert in causality. I am familiar with causality, but I don't follow its research closely.
Questions for authors
Q1. OOD training. Line 451 writes, "we exclude one feature k_OOD during training". How exactly is this done? Are nodes corresponding to that feature deleted from the input graphs during training? Or is the loss function modified to ignore that variable. Please clarify.
Q2. OOD experiment. What is the target variable in each of table 3's rows? Was the the feature column also the target?
Q3. Conversion to point estimates. Please elaborate the exact procedure of reducing the mixture outputs to the point estimates reported in the tables.
4 comments
Thank you for the revision. Most of my concerns have been addressed, but a few remain, as detailed below.
W1b Untested claim
Review: Another related experiment should compare CFGNP against itself on a given causal graph. For example, restrict message passing to the true or estimated edges rather than the full graph, to show what the missing graph costs or gains.
Unresolved. The authors did not conduct this experiment.
W2. CANDLE
Partially resolved.
- Preservation can be gamed by copying. A model that returns the input unchanged would score close to the oracle ceiling on preservation and close to zero on effectiveness. The authors mention that their method "trades effectiveness for preservation". However, copying does the same. Hence, this experiment needs a copy baseline to put CFGPN's numbers in context. Please add the copy baseline to Table 7, as already done for CheXpert in Table 4.
- Line 349 of the current revision writes, "CFGNP trades effectiveness for preservation, performing as well as the baselines." I'd revise it, as DeepBC wins on both effectiveness and composition.
Note.
W2. It will allow the authors to measure whether the generated image changes the intended factor while preserving the others.
In hindsight, my request for a preservation metric assumes interventional semantics while the paper uses backtracking counterfactuals. Interventional counterfactuals hold upstream variables fixed while backtracking counterfactuals do not. So a backtracking counterfactual making an object large could legitimately change an indoor scene to an outdoor one (a non-target attribute change), since the confounder links size and scene. I'd welcome a brief comment on whether full preservation is the right target for backtracking counterfactuals.
Connected to this line of thought, the authors mention in App. E.5:
In the causal graph of CANDLE the factors do not cause each other (they are related only through confounders) and all of them cause the image. Changing one factor should therefore change exactly that factor in the image and leave all other factors unchanged, which makes the correctness of a counterfactual image measurable.
Since CFGNP predicts backtracking counterfactuals, under which confounders may change and legitimately alter non-target factors, preservation of all non-target factors may not the right target.
W5. Uncertainty.
The reported metrics are limited to MSE, ROC-AUC and accuracy; no uncertainty metrics (e.g. calibration error or negative log-likelihood) are reported, and there is no experiment demonstrating that the predicted uncertainty is meaningful or useful. The authors should evaluate uncertainty directly.
Unresolved: The authors have not reported the uncertainty estimates of the datasets used in the experiments.
They could also show how predictive uncertainty increases when the underlying SCM is less identifiable from the observed data, or its decrease with increasing observed data.
Partially resolved: The authors have somewhat addressed the last part (decreasing uncertainty with increasing observed data). However, they have tested it on a two-node SCM, which is too simplistic. They should present the uncertainty change on the actual datasets used in the paper.
All remaining weaknesses are resolved. W6 (OOD experiments): resolved by the authors retracting the claim and removing it from the paper.
PS. Interpreting Table 10 on page 43 requires comparing numbers from Table 1 on page 8. For the reader's convenience, add a copy of Table 1 alongside Table 10 in the appendix, in a similar format.
W1b.
We thank the reviewer for the comment and apologize for the confusion. We had already included it in the rebuttal (Table 2, p.9) but forgot to mention that in the reply to the comment. Restricting Message-Passing to the true causal graph (with observations) reduced performance across 6 out of 7 datasets compared to normal CFGNP. In one it even performed worse than CFGNP without context. However, based on reviewer Nd2t’s comment, we have now supplied the ground-truth to CFGNP via edge labels. This keeps Proposition A.1 and improves performance. Now, CFGNP with observational context and true causal graph for instance performs best on German loan, Mshape_LIN and Mshape_NADD and is within on std of the best results on the other datasets. CFGNP without observational context but with true graph now performs best on Triangle_LIN and Triangle_NADD. Vanilla CFGNP performs best in the rest. We have updated Table 2 (p.9) accordingly. We have also included an ablation between the previous and the new label-based approach in section E.12 (p.44).
W2
We thank the reviewer for their comment. We have added a copy baseline, both one that just copies the image and one that copies the hidden state inside a VAE (because this is what CFGNP uses). For all values, see Table 8 (Sec. E.5, p.40). The VAE-copy has considerably lower effectiveness with 0.069 compared to CFGNPs 0.316. Preservation is slightly higher (0.765 vs. 0.694) and composition MSE is almost 0 (0.0010) as expected. We have updated the manuscript to: “CFGNP has the highest preservation; DeepBC has higher effectiveness and better composition.” (p.7, l.348)
Note Reply.
We thank the reviewer for this point. We agree that under backtracking semantics, non-target factors that share a confounder with the intervened factor may legitimately change. In CANDLE, however, this dependence is limited. We measured it for all pairs of factor labels using Cramér’s V with a permutation test (Table 9, App. E.5, p.40). Only four of the 15 pairs are dependent, all with moderate strength (largest bias-corrected V=0.36), and rotation and light are independent of all other factors. Since a backtracking counterfactual can change a non-target factor only through a shared confounder, interventional and backtracking counterfactuals agree on the non-target factors for most interventions, and preservation measures the same quantity under both semantics. Moreover, on CANDLE we train CFGNP with the interventional target (intervened factor changed, all other factors unchanged), so preservation is the objective the model is trained for.
W5.
We thank the reviewer for their comment. We have added Table 10 (App. E.8, p.42) that reports the calibration error as the uncertainty metric. We compare to all baselines where we use counterfactual supervision with an NLL loss (Details in App. D, p.32). Additionally we compare to vanilla VACA and VCI as they also produce probability distributions. CFGNP has the lowest calibration error on German loan. Otherwise, especially vanilla CFGNP has a lower error. VCI sometimes has a lower calibration error. The other two baselines with NLL supervision have considerably higher calibration error. As both VCI and VACA have the purpose of producing probability distributions, they are also developed to have low calibration errors.
However, they have tested it on a two-node SCM, which is too simplistic.
We chose the two-node SCM because it is a setting in which the true counterfactual distribution given the observed context can be computed analytically. This allows us to verify that the predicted uncertainty matches the correct uncertainty, not only that it decreases. As requested, we now also report the change in uncertainty on all seven benchmark datasets (Fig. 25, p.43). We train CFGNP on 100 observations and then evaluate it only on an observational context form 1 to 100. We report the predicted std, the 90% interval coverage and the NLL. On all datasets, the predicted standard deviation decreases monotonically with context size and levels off at a dataset-dependent floor, highest for the NADD case. At the training context size, coverage is close to nominal (0.87–0.97). Since the models are trained with 100 context samples, they are overconfident for very small contexts, which lie outside the training regime.
PS. Interpreting Table 10 on page 43 requires comparing numbers from Table 1 on page 8. For the reader's convenience, add a copy of Table 1 alongside Table 10 in the appendix, in a similar format.
We have attached the copy of Table 1 alongside Table 10.
Thank you for the response. It addresses my remaining concerns. I am raising my rating from weak acceptance (3) to clear acceptance (4).
We thank the reviewer for their comments. When incorporating the comments, we reran old experiments as well to ensure that all methods were trained on exactly the same dataset. Since the construction is stochastic, results have changed in many cases. Based on the previous std of the loss, it seems that some seeds had many outliers. This is not the case anymore.
W1: Untested Claim. We thank the reviewer for their comment. We use NOTEARS to get the causal graph and subsequently ran the baselines on the sampled causal graphs. Results have gotten considerably worse, for instance BGM from 4.52 × 10^−4 MSE to 4.32 × 10^−3 on triangle_LIN. The exception is triangle_NADD where results got better, even than CFGNP. Full results are in App. E.10 (p.42-43), referred to on l.364.
W2: Chexpert lacks ground-truth counterfactuals. We thank the reviewer for this suggestion and have added experiments on CANDLE. For each test image, we intervene on each of the six factors in turn, setting it to a new random value. A ResNet-18 factor classifier (0.82 test accuracy) reads the factors from the generated images, and we report three metrics: Effectiveness: the intervened factor takes its new value. Preservation: all other factors stay the same. Composition: the reconstruction MSE under a null intervention.
We also run VCI and DeepBC on the same split, using a shared encoder/decoder. App. E.5 (p.39, referred to in l.349) shows the results. CFGNP trades effectiveness for preservation, performing as well as DeepBC (lower effectiveness but much higher preservation). VCI performs worse.
W3: VCI on synthetic benchmarks VCI is now included in the synthetic experiments (Table 1, p.8). VCI performs considerably worse than the other baselines and CFGNP. For instance, for german loan its MSE is only 1.95 × 10^−1.
W4: Context-size and sample-efficiency ablation in the main evaluation. We used a CFGNP pretrained on 100 observations for inference on smaller context sizes {1, 5, 10, 25, 50, 75, 100}. Results get worse on smaller context sizes (App. E.7, p.41).
W5: Uncertainty. We thank the reviewer for their suggestion. We added a synthetic experiment evaluating CFGNP on increasing observational context sizes, measuring the predictive std and the KL divergence. The predictive std is close to the analytically computed Bayes-optimum. Both go down with increasing context size and approach the predictive std of the true counterfactual mechanism. The KL divergence between CFGNP and the Bayes-optimal decreases slightly with increasing context size, possibly because the model has more issues dealing with noise for small sample sizes. More importantly, the KL divergence between the true counterfactual distribution and CFGNP is much bigger for small context sizes and approaches the other KL divergence for larger contexts. This shows that CFGNP does quantify the increased uncertainty for smaller observational context, i.e. when the causal graph is less identifiable from observational data. Details are in App. E.6 (p.40) and in the main results section (paragraph “Predictive Uncertainty”, p.9, l.482-).
W6: OOD experiment. We thank the reviewer for scrutinising the OOD experiment. On closer inspection, the results do not support our claim that CFGNP generalizes to variables unseen during training, as the models seem to have learned to just copy the input. We have therefore removed this claim from the abstract, introduction and conclusion, and report the experiment in App. E.1.2 (p.34) as a limitation. The issue is specific to this setting and does not affect the other results. In Table 4 (p.34) we have added a baseline that just predicts the input. After splitting the features into changed features (the intervention index) and preserved features (the rest), the scores of the baseline and the OOD models are very similar, except for the age feature.
W7: Stale references. We have updated the references to the published versions.
Q1: OOD Training We now state in the document that the excluded feature k_OOD is removed by deleting the corresponding node from the graph during training (p.34, l. 1813).
Q2: OOD Experiment We have updated the table to only show changed features and preserved features (now Table 4, p.34).
Q3: Conversion to point estimates During training we use all outputs of the model to additionally obtain the variance, which we can use to quantify uncertainty. For the reported scores during evaluation, we use the weighted sum of μ and w (App. B.5, p.25, l. 1325).