- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper explores counterfactual image editing under a structural causal model. Given an image, its attribute factors and a causal graph, the goal is to produce a countefactual image under do(action), while changing descendants of A and preserve its non-descendants. The paper proposes MSMS, a method for selecting among counterfactual candidates using a pretrained generator as extra evidence.
Critical strengths
None
Critical weaknesses
- The main text is extremely dense and heavy on notation, and is not self-contained. Overall, a fairly simple pipeline is presented with heavy formal machinery which is very difficult to parse. Many essential facts for interpreting the method appear only in the appendix (which is 40+ pages long). For example, the fact that correction queries read only the target (E.4), and that CBMNIST correction scores are identically zeros (E.3). And the appendix reads like implementation logs even more that the main paper.
- The paper's motivation is that admissible proxies differ in how they propagate an intervention to descendants (Fig. 1, section 3.3). However, the reported correction queries read only the intervened target (E.4, F.3), so the anchor never informs the descendant mechanisms. Differences in correction scores can then come only from the learned scoring renderer, not from the ambiguous causal mechanisms. This fact breaks the motivation.
Questions for authors
- Since the reported correction queries read only the target, which by Lemma 1 is identical for all graph-compatible proxies, what information about the descendant mechanisms does the selection actually use? Can you report results with correction queries on descendants, and explain why DR is worse than BD-CLS on CBMNIST and ANCM on SDXL?
- Why evaluate on faces generated by the anchor itself rather than on real labeled faces such as CelebA-HQ?
6 comments
Results with correction queries on descendants
Proposition 4 (Appendix D.12) splits the condition of Equation (6) into a query condition, that differences in the queried responses control the edit risk, and an anchor condition, that the anchor's own edits lie close to the true counterfactual responses. We use this split to read the results below.
Correction queries on descendants, submitted face edges. We ran correction queries that leave the descendant open in the anchor's prompt and compare candidates that share the observational law but differ in the counterfactual descendant law (monotone, independent, antitone and mixtures, plus factual copy). On do(Young) with Smiling open (Stable Diffusion XL, 400 sources, 16 draws), the anchor's answer depends on both the source and the intervention. Against a no-intervention control, the Smiling-present cells move from 0.31 to 0.68 and from 0.70 to 0.43. Open queries therefore make the descendant comparison nontrivial. On held-out sources the correction score of the factual-preserving coupling exceeded that of the selected coupling by 0.012 [−0.004, 0.029], so the preference is not established there. The preferred coupling also moved with the editing strength, as Weakness 1 of Reviewer T2ym anticipates, since 0.50, 0.70 and 0.85 gave the monotone, independent and antitone couplings.
Anchor condition on the same edges. On a second Stable Diffusion XL face bank whose declared structural causal model makes the independent coupling true, selection did not recover it. On Young to Smiling it chose the monotone coupling (bootstrap share 0.962), with counterfactual-law error 0.082 against 0.063 for a random feasible candidate. Its open answers strongly track factual Smiling, unlike the declared independent response law. The heavy-makeup edge cannot test the anchor, because its open prompt coincided with the value-0 prompt. This anchor therefore does not meet the anchor condition for descendants on the Young-to-Smiling edge, and the revision reports both edges and the prompt diagnosis.
Summary. For the submitted face anchors, open descendant queries make the anchor's comparison informative about its own preference, and that preference depends on the editing procedure. The revision reports these results with the conditions of Proposition 4.
Question 1, ColoredBar-MNIST trained in one world
Comment 1 explains that the ColoredBar-MNIST rows of Table 1 were not trained in one world. We have now trained all five methods again in one world, the fixed corpus with the bias-0.75 width mechanism, with three training seeds per method. We evaluated them on the ColoredBar-MNIST bank of Table 1 with the same attribute predictor, descendant response discrepancy and Fréchet Inception distance. Our row is the final checkpoint of observational training without correction or selection, so it does not test the selection rule. It also differs from our Table 1 row in the predictor used during training, the number of training steps, the number of candidates and the weight of the stable term.
| Method | Target success ↑ | Non-descendant drift ↓ | Descendant response discrepancy ↓ | Fréchet Inception distance ↓ |
|---|---|---|---|---|
| Ours, observational-only | 0.982 ± 0.005 | 0.011 ± 0.001 | 0.569 ± 0.016 | 12.9 |
| BD-CLS | 0.975 ± 0.012 | 0.011 ± 0.003 | 0.399 ± 0.025 | 21.4 |
| ANCM | 1.000 ± 0.000 | 0.020 ± 0.005 | 0.857 ± 0.085 | 42.2 |
| Counterfactual generative networks | 0.463 ± 0.024 | 0.350 ± 0.038 | 0.802 ± 0.029 | 152.4 |
| CausalDiffAE | 0.482 ± 0.049 | 0.171 ± 0.057 | 0.478 ± 0.010 | 7.6 |
The spreads are sample standard deviations over training seeds, whereas the baseline spreads in Table 1 are over evaluation seeds. BD-CLS (Pan and Bareinboim, 2025) keeps the lowest descendant response discrepancy, as in Table 1, and CausalDiffAE also has a lower discrepancy than our row. Our row has the same drift as BD-CLS and a lower Fréchet Inception distance. On this bank ANCM (Pan and Bareinboim, 2024) has the highest target success.
Our baseline runs are adaptations, and six of their choices matter for reading the results. For the counterfactual generative networks (Sauer and Geiger, 2021) and for BD-CLS on Stable Diffusion XL, the requested factors change only the intervened factor, so descendants keep their factual values. On the face benchmarks the counterfactual generative network is built on the frozen StyleGAN3 generator. In the single-world ColoredBar-MNIST runs, ANCM uses a graph with a latent confounder between digit and digit color that the data mechanism does not have. Its training objective also differs from that of Pan and Bareinboim (2024), an evidence lower bound plus a cross-entropy term that matches P(V | I). CausalDiffAE writes the requested value into the exogenous coordinate of the intervened variable rather than setting the causal latent itself (Algorithm 2, line 4, of Komanduri et al., 2024), and it does not read the reference counterfactual. On ColoredBar-MNIST, BD-CLS renders from new noise instead of noise inferred from the source image (Algorithm 1, line 3, of Pan and Bareinboim, 2025).
We also ran versions closer to the original methods on the same bank. ANCM retrained with the objective of its paper gives target success 0.996, drift 0.001, discrepancy 0.533 and Fréchet Inception distance 37.6. CausalDiffAE with the causal latent set to the requested value gives 0.611, 0.108, 0.520 and 4.4. BD-CLS with source noise estimated by an approximate inversion of its diffusion sampler gives 0.975, 0.005, 0.420 and 23.9. Counterfactual generative networks with labels supplied per mechanism give 0.343, 0.448, 0.740 and 156.9. ANCM with its original objective, CausalDiffAE and BD-CLS in these versions all have a lower discrepancy than our row. On a second bank of the same world with 16 draws per query, where the discrepancy compares the distribution of the 16 descendant readings with the stored reference value, ANCM with its original objective has a lower discrepancy than our row, by 0.016 [0.004, 0.028] over sources, and for CausalDiffAE the difference of 0.034 [−0.013, 0.080] includes zero. BD-CLS with source noise estimated by inversion has a lower discrepancy than our row there as well, by 0.070 [0.039, 0.099], together with a higher target success and a lower drift. The revision keeps Table 1 and adds these results to Appendix E.
What the selection uses and the discrepancy comparisons
Thank you for the critical reading. We answer each point directly and Question 2 in comment 4.
Weakness 1, density. We agree. The pipeline is simple to state. Among candidates that fit the observations equally well, MSMS chooses the proxy whose query responses shift least from those of the anchor under the stable constraint, and the edit renders that proxy's counterfactual factors with the anchor. The formal statements serve the guarantees of Theorems 1 and 2, and we keep them in Section 3 because Section 5 and the proofs refer to them. To make the main text self-contained, Section 3.2 of the revision states the two facts the reviewer mentioned, that correction queries read only the target and that the correction scores are zero on ColoredBar-MNIST (Pan and Bareinboim, 2024, 2025). Section 3.1 defines factor abduction, factor intervention and rendering where the factorization of the edit introduces them, and Section 3.2 defines the anchor as a pretrained generator with its editing procedure. Appendix E.12 adds a walk-through of one query, a table of the anchors' editing procedures and a component table, and Section 3.2 points to it (our answer to Reviewer ZQG2). We also agree that the appendix reads like implementation logs, and we will reorganize it into supporting sections for the final version.
Weakness 2 and Question 1, what descendant information the selection uses. With the submitted target-only correction queries, the score does not compare descendants with the anchor. Descendant values reach it only by shifting the predictor's reading of the target in an image rendered from the candidate's full factor rollout, and our answer to Reviewer T2ym names a target reading that does not respond to the descendants as a precondition of selection. A direct comparison of descendants with the anchor needs queries that leave the descendant open. These queries directly compare descendant responses, as required to assess the query condition of Proposition 4 (Appendix D.12) for the descendant mechanisms, and comment 2 reports them. On ColoredBar-MNIST the correction scores are zero and selection reduces to the stable rule, as Appendix E.3 states.
Question 1, the discrepancy against BD-CLS on ColoredBar-MNIST. The rows were not trained in one data world. Our candidates used a version in which bar width is independent of bar color. BD-CLS (Pan and Bareinboim, 2025) and CausalDiffAE (Komanduri et al., 2024) used a fixed corpus with a width mechanism of bias 0.75, and ANCM (Pan and Bareinboim, 2024) and counterfactual generative networks (Sauer and Geiger, 2021) were trained on data generated procedurally from the same factor mechanism. The evaluation reference follows the bias-0.75 mechanism, under which every query requires the descendant to change, and in 76% of our bar-color edits the bar width read from the image kept its factual value. The revision marks this row in Table 1 and reports all five methods trained in one world in Appendix E (comment 3).
Question 1, the discrepancy against ANCM on Stable Diffusion XL. Our method has 0.370 against 0.276 for ANCM, a difference of 0.093 [0.032, 0.155] over sources, and 0.402 against 0.255 under a CelebA classifier (Liu et al., 2015). For both methods most of the discrepancy is under-response, where the reference changes a descendant and the edited image keeps the factual value. Within our method, the discrepancy concentrates in the 815 of 2,400 images where the pretrained pipeline did not render the requested target (0.75, against 0.17 on the 1,585 where it did), while ANCM has 0.37 on its 957 realized images and 0.21 on the other 1,443. The groups differ in composition, so this split describes where the discrepancy is concentrated within our outputs. ANCM's outputs on this benchmark are blurred faces (Fréchet Inception distance 180.2), and keeping every factual descendant value would give 0.620.