acceptodds
Under review as a conference paper at ICLR 2027

Beyond Feasibility: Minimal-Shift Proxy Selection for Counterfactual Image Editing

Abstract

Counterfactual image editing asks how a given image would change under an intervention on a semantic attribute, accounting for causal effects while preserving unaffected attributes. Observational data and a causal graph cannot generally determine this change uniquely, as observationally equivalent models may imply different counterfactual images. Existing methods characterize admissible solutions within partial identification bounds but leave their selection unresolved, and their implementations can require full generative training or image–factor invertibility. To address these limitations, we propose Minimal-Shift Mechanism Selection (MSMS), which uses a large pretrained generator as additional evidence for ranking counterfactually consistent mechanisms. MSMS favors candidates whose editing responses most closely match those of the pretrained model, subject to preserving unaffected attributes. We formulate this comparison in a shared space across heterogeneous architectures and establish sufficient conditions for -consistency under realizability and reference calibration. By separating factor abduction from image rendering, MSMS reuses pretrained rendering capabilities without requiring an invertible mapping between factors and images, supporting diffusion models and GANs. Experiments across three benchmarks demonstrate MSMS's ability to perform targeted edits while preserving non-descendant attributes. We further examine how pretrained responses guide the selection of candidate causal models.

Primary area:
generative models
Keywords:
Causal Inference, Counterfactual Editing, Generative Models, Structural Causal Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_H12iOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores counterfactual image editing under a structural causal model. Given an image, its attribute factors and a causal graph, the goal is to produce a countefactual image under do(action), while changing descendants of A and preserve its non-descendants. The paper proposes MSMS, a method for selecting among counterfactual candidates using a pretrained generator as extra evidence.

Critical strengths

None

Critical weaknesses
  1. The main text is extremely dense and heavy on notation, and is not self-contained. Overall, a fairly simple pipeline is presented with heavy formal machinery which is very difficult to parse. Many essential facts for interpreting the method appear only in the appendix (which is 40+ pages long). For example, the fact that correction queries read only the target (E.4), and that CBMNIST correction scores are identically zeros (E.3). And the appendix reads like implementation logs even more that the main paper.
  2. The paper's motivation is that admissible proxies differ in how they propagate an intervention to descendants (Fig. 1, section 3.3). However, the reported correction queries read only the intervened target (E.4, F.3), so the anchor never informs the descendant mechanisms. Differences in correction scores can then come only from the learned scoring renderer, not from the ambiguous causal mechanisms. This fact breaks the motivation.
Questions for authors
  1. Since the reported correction queries read only the target, which by Lemma 1 is identical for all graph-compatible proxies, what information about the descendant mechanisms does the selection actually use? Can you report results with correction queries on descendants, and explain why DR is worse than BD-CLS on CBMNIST and ANCM on SDXL?
  2. Why evaluate on faces generated by the anchor itself rather than on real labeled faces such as CelebA-HQ?
6 comments
AuthorsOpenReview ↗

What the selection uses and the discrepancy comparisons

Thank you for the critical reading. We answer each point directly and Question 2 in comment 4.

Weakness 1, density. We agree. The pipeline is simple to state. Among candidates that fit the observations equally well, MSMS chooses the proxy whose query responses shift least from those of the anchor under the stable constraint, and the edit renders that proxy's counterfactual factors with the anchor. The formal statements serve the guarantees of Theorems 1 and 2, and we keep them in Section 3 because Section 5 and the proofs refer to them. To make the main text self-contained, Section 3.2 of the revision states the two facts the reviewer mentioned, that correction queries read only the target and that the correction scores are zero on ColoredBar-MNIST (Pan and Bareinboim, 2024, 2025). Section 3.1 defines factor abduction, factor intervention and rendering where the factorization of the edit introduces them, and Section 3.2 defines the anchor as a pretrained generator with its editing procedure. Appendix E.12 adds a walk-through of one query, a table of the anchors' editing procedures and a component table, and Section 3.2 points to it (our answer to Reviewer ZQG2). We also agree that the appendix reads like implementation logs, and we will reorganize it into supporting sections for the final version.

Weakness 2 and Question 1, what descendant information the selection uses. With the submitted target-only correction queries, the score does not compare descendants with the anchor. Descendant values reach it only by shifting the predictor's reading of the target in an image rendered from the candidate's full factor rollout, and our answer to Reviewer T2ym names a target reading that does not respond to the descendants as a precondition of selection. A direct comparison of descendants with the anchor needs queries that leave the descendant open. These queries directly compare descendant responses, as required to assess the query condition of Proposition 4 (Appendix D.12) for the descendant mechanisms, and comment 2 reports them. On ColoredBar-MNIST the correction scores are zero and selection reduces to the stable rule, as Appendix E.3 states.

Question 1, the discrepancy against BD-CLS on ColoredBar-MNIST. The rows were not trained in one data world. Our candidates used a version in which bar width is independent of bar color. BD-CLS (Pan and Bareinboim, 2025) and CausalDiffAE (Komanduri et al., 2024) used a fixed corpus with a width mechanism of bias 0.75, and ANCM (Pan and Bareinboim, 2024) and counterfactual generative networks (Sauer and Geiger, 2021) were trained on data generated procedurally from the same factor mechanism. The evaluation reference follows the bias-0.75 mechanism, under which every query requires the descendant to change, and in 76% of our bar-color edits the bar width read from the image kept its factual value. The revision marks this row in Table 1 and reports all five methods trained in one world in Appendix E (comment 3).

Question 1, the discrepancy against ANCM on Stable Diffusion XL. Our method has 0.370 against 0.276 for ANCM, a difference of 0.093 [0.032, 0.155] over sources, and 0.402 against 0.255 under a CelebA classifier (Liu et al., 2015). For both methods most of the discrepancy is under-response, where the reference changes a descendant and the edited image keeps the factual value. Within our method, the discrepancy concentrates in the 815 of 2,400 images where the pretrained pipeline did not render the requested target (0.75, against 0.17 on the 1,585 where it did), while ANCM has 0.37 on its 957 realized images and 0.21 on the other 1,443. The groups differ in composition, so this split describes where the discrepancy is concentrated within our outputs. ANCM's outputs on this benchmark are blurred faces (Fréchet Inception distance 180.2), and keeping every factual descendant value would give 0.620.

AuthorsOpenReview ↗

Results with correction queries on descendants

Proposition 4 (Appendix D.12) splits the condition of Equation (6) into a query condition, that differences in the queried responses control the edit risk, and an anchor condition, that the anchor's own edits lie close to the true counterfactual responses. We use this split to read the results below.

Correction queries on descendants, submitted face edges. We ran correction queries that leave the descendant open in the anchor's prompt and compare candidates that share the observational law but differ in the counterfactual descendant law (monotone, independent, antitone and mixtures, plus factual copy). On do(Young) with Smiling open (Stable Diffusion XL, 400 sources, 16 draws), the anchor's answer depends on both the source and the intervention. Against a no-intervention control, the Smiling-present cells move from 0.31 to 0.68 and from 0.70 to 0.43. Open queries therefore make the descendant comparison nontrivial. On held-out sources the correction score of the factual-preserving coupling exceeded that of the selected coupling by 0.012 [−0.004, 0.029], so the preference is not established there. The preferred coupling also moved with the editing strength, as Weakness 1 of Reviewer T2ym anticipates, since 0.50, 0.70 and 0.85 gave the monotone, independent and antitone couplings.

Anchor condition on the same edges. On a second Stable Diffusion XL face bank whose declared structural causal model makes the independent coupling true, selection did not recover it. On Young to Smiling it chose the monotone coupling (bootstrap share 0.962), with counterfactual-law error 0.082 against 0.063 for a random feasible candidate. Its open answers strongly track factual Smiling, unlike the declared independent response law. The heavy-makeup edge cannot test the anchor, because its open prompt coincided with the value-0 prompt. This anchor therefore does not meet the anchor condition for descendants on the Young-to-Smiling edge, and the revision reports both edges and the prompt diagnosis.

Summary. For the submitted face anchors, open descendant queries make the anchor's comparison informative about its own preference, and that preference depends on the editing procedure. The revision reports these results with the conditions of Proposition 4.

AuthorsOpenReview ↗

Question 1, ColoredBar-MNIST trained in one world

Comment 1 explains that the ColoredBar-MNIST rows of Table 1 were not trained in one world. We have now trained all five methods again in one world, the fixed corpus with the bias-0.75 width mechanism, with three training seeds per method. We evaluated them on the ColoredBar-MNIST bank of Table 1 with the same attribute predictor, descendant response discrepancy and Fréchet Inception distance. Our row is the final checkpoint of observational training without correction or selection, so it does not test the selection rule. It also differs from our Table 1 row in the predictor used during training, the number of training steps, the number of candidates and the weight of the stable term.

MethodTarget success ↑Non-descendant drift ↓Descendant response discrepancy ↓Fréchet Inception distance ↓
Ours, observational-only0.982 ± 0.0050.011 ± 0.0010.569 ± 0.01612.9
BD-CLS0.975 ± 0.0120.011 ± 0.0030.399 ± 0.02521.4
ANCM1.000 ± 0.0000.020 ± 0.0050.857 ± 0.08542.2
Counterfactual generative networks0.463 ± 0.0240.350 ± 0.0380.802 ± 0.029152.4
CausalDiffAE0.482 ± 0.0490.171 ± 0.0570.478 ± 0.0107.6

The spreads are sample standard deviations over training seeds, whereas the baseline spreads in Table 1 are over evaluation seeds. BD-CLS (Pan and Bareinboim, 2025) keeps the lowest descendant response discrepancy, as in Table 1, and CausalDiffAE also has a lower discrepancy than our row. Our row has the same drift as BD-CLS and a lower Fréchet Inception distance. On this bank ANCM (Pan and Bareinboim, 2024) has the highest target success.

Our baseline runs are adaptations, and six of their choices matter for reading the results. For the counterfactual generative networks (Sauer and Geiger, 2021) and for BD-CLS on Stable Diffusion XL, the requested factors change only the intervened factor, so descendants keep their factual values. On the face benchmarks the counterfactual generative network is built on the frozen StyleGAN3 generator. In the single-world ColoredBar-MNIST runs, ANCM uses a graph with a latent confounder between digit and digit color that the data mechanism does not have. Its training objective also differs from that of Pan and Bareinboim (2024), an evidence lower bound plus a cross-entropy term that matches P(V | I). CausalDiffAE writes the requested value into the exogenous coordinate of the intervened variable rather than setting the causal latent itself (Algorithm 2, line 4, of Komanduri et al., 2024), and it does not read the reference counterfactual. On ColoredBar-MNIST, BD-CLS renders from new noise instead of noise inferred from the source image (Algorithm 1, line 3, of Pan and Bareinboim, 2025).

We also ran versions closer to the original methods on the same bank. ANCM retrained with the objective of its paper gives target success 0.996, drift 0.001, discrepancy 0.533 and Fréchet Inception distance 37.6. CausalDiffAE with the causal latent set to the requested value gives 0.611, 0.108, 0.520 and 4.4. BD-CLS with source noise estimated by an approximate inversion of its diffusion sampler gives 0.975, 0.005, 0.420 and 23.9. Counterfactual generative networks with labels supplied per mechanism give 0.343, 0.448, 0.740 and 156.9. ANCM with its original objective, CausalDiffAE and BD-CLS in these versions all have a lower discrepancy than our row. On a second bank of the same world with 16 draws per query, where the discrepancy compares the distribution of the 16 descendant readings with the stored reference value, ANCM with its original objective has a lower discrepancy than our row, by 0.016 [0.004, 0.028] over sources, and for CausalDiffAE the difference of 0.034 [−0.013, 0.080] includes zero. BD-CLS with source noise estimated by inversion has a lower discrepancy than our row there as well, by 0.070 [0.039, 0.099], together with a higher target success and a lower drift. The revision keeps Table 1 and adds these results to Appendix E.

Reviewer_ZQG2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes MSMS, a methodology used for causal counterfactual editing. The framework chooses the best option amongst multiple causally admissible models, through a scoring system on the performed edits that is based on a pretrained generator. Experiments are conducted on three benchmarks (ColoredBar-MNIST, SDXL- and StyleGAN-generated faces) using both diffusion and GAN generators on four new metrics proposed by the authors for evaluation, along with fidelity metrics.

Critical strengths
  • The problem is well-posed and causal consistency in counterfactual edits is extremely important. It enforces that edits are not made from biased correlations in neural networks, but instead are informed and directly related to what actually changes the content
  • The authors make a great effort to be transparent and the methodology is largely adaptable, functioning in different types of image generators
Critical weaknesses
  • The methodology is relatively hard to follow and I feel that readers not familiar with causal models and their notation will have a hard time following. Section 3 runs very long and it is not entirely clear what the intended outcome is. From what I understand, the entire point of the framework is to choose a proxy which provides the minimal shift from the original anchor, but up until that point there is no example of what a query, a proxy or an anchor is for the utilize datasets. For someone to understand how these translate on an architectural basis they need to dig through the appendix, this is information that should be explicitly state in the main text along with the definitions and most of the math moved into the appendix for anyone to check. It is also very word heavy with missing definitions of terms, such as "factor intervention" in line 169, "factor abduction" in the abstract and the word "rendering" being used for image generation.

  • The evaluation does not reflect what the methodology states to do. Table 1 is not indicative that the model appears to be targeting counterfactual parameters and all metrics, apart from FID are not cited and are newly introduced. They also mostly favour the proposed methodology, while FID only succeeds on SDXL with CBMNIST and StyleGAN, especially, showcasing that the distributions drifts significantly from the intended.

Questions for authors
  • Could you explain further the discrepancies in the metrics? What causes large FID differences in certain models? Are there any other established metrics that could be utilized for this, such as MMD? What is the accuracy of the attribute predictor used, and does TS hold under an externally trained classifier?
  • Could you state concretely what the anchor, proxy, and stable/correction queries are (which network, inputs and outputs), and what ‘factor abduction’ and ‘factor intervention’ mean operationally as architectural components?
5 comments
AuthorsOpenReview ↗

Operational specification and evaluation

Thank you for the detailed comments. We answer Question 2 first, as it also addresses readability.

Question 2, what each component is. Take a StyleGAN3 (Karras et al., 2021) face, its observed factors and the request do(Male:=0).

  • Proxy. One graph-constrained multilayer perceptron per factor. It takes the factual factor tuple and an intervention and returns a counterfactual factor tuple. Candidates are its saved training iterates.
  • Factor abduction. A learned encoder maps the factual factor tuple to a background code.
  • Factor intervention. Male is set to 0, the descendant Eyeglasses is recomputed by its mechanism from the background code, and Young and Smiling keep their factual values.
  • Rendering. The resulting factors and the source image go to the pretrained editing pipeline, which returns the edited image. On Stable Diffusion XL (Podell et al., 2024) this is image-to-image editing in the manner of stochastic differential editing (Meng et al., 2022) at strength 0.70 with a prompt built from the factor values. On StyleGAN3 it is encoder-based inversion (Tov et al., 2021) into W+ followed by attribute-direction edits.
  • Anchor. The same pretrained pipeline applied directly to the source with the requested target.
  • Stable query. A source and an intervention. The predictor reads the non-descendants (Young, Smiling) of the candidate's response and compares them with the source.
  • Correction query. The same input. It compares the candidate's response on the intervened target with the anchor's edit of the same source. During training and selection on faces the candidate's factors are rendered by a learned scoring renderer, a multilayer perceptron followed by a convolutional decoder. The revision adds this example and a component table to the appendix, with each step defined as above, and points to them from Section 3.

Question 1, the metrics and the large Fréchet Inception distance differences. Target success asks whether the requested target changed, non-descendant drift whether non-descendants stayed fixed, and descendant response discrepancy how descendants responded. Target success corresponds to the effectiveness criterion of prior evaluations of counterfactual image models (Monteiro et al., 2023). For the Fréchet Inception distance (Heusel et al., 2017), how each method produces an image gives a plausible explanation of its value, which we have not isolated in a controlled comparison. ANCM (Pan and Bareinboim, 2024) trains its own generator from the benchmark images and yields blurred faces with the highest distance on both face benchmarks (180.2 and 158.8). Counterfactual generative networks (Sauer and Geiger, 2021), built here on the frozen StyleGAN3 generator, generate a new face without conditioning on the source and have the lowest distance on StyleGAN3 (15.0) together with a drift of 0.204 against our 0.098. Our StyleGAN3 editing path is deterministic given the factors, so draws with the same factors give the same image, which lowers sample diversity and plausibly raises the distance. On maximum mean discrepancy (Gretton et al., 2012), Table 8 reports the kernel Inception distance (Bińkowski et al., 2018), a squared maximum mean discrepancy between Inception features.

Question 1, predictor accuracy and a CelebA classifier. On both face benchmarks one attribute predictor supplies the selection measurements and the final scores (validation accuracy 1.000 on all five Stable Diffusion XL attributes). Section 5 (line 376) calls it independent and the opening of Appendix E separate, which is incorrect, and the revision corrects both passages and Appendix E.5. We rescored all 18,400 saved face renders of Table 1 with a classifier trained on CelebA (Liu et al., 2015), with validation accuracy 0.905 to 0.998. The same classifier screened the generated training images and the StyleGAN3 evaluation sources (Appendix E.6). The target-success order of the methods is unchanged on both face benchmarks. On Stable Diffusion XL our margin over BD-CLS (Pan and Bareinboim, 2025) is 0.150 [0.066, 0.234] under the CelebA classifier and 0.040 [−0.061, 0.142] under the original predictor, over the two training seeds both methods share.

AuthorsOpenReview ↗

Addressing Weakness 1 & 2

Weakness 1, readability. We agree. The intended outcome of Section 3 is the choice of one proxy among candidates that fit the observations equally well, namely the one whose query responses shift least from those of the anchor under the stable constraint. The edit is then that proxy's counterfactual factors rendered by the anchor. The revision defines factor abduction, factor intervention and rendering in Section 3.1, where the factorization of the edit introduces them, and defines the anchor as a pretrained generator with its editing procedure in Section 3.2, next to the example of a stable and a correction query there. It adds to Appendix E.12 a walk-through of one query with the proxy, the anchor and the queries (the example in the comment above), a table of the anchors' editing procedures and a component table, and Section 3.2 points to them. We keep the formal statements in Section 3, since Section 5 and the proofs refer to them.

Weakness 2, what the evaluation measures. As answered in the above comment regarding Question 1, target success checks that the intervened factor took the requested value, and non-descendant drift checks that the factors outside the intervention's descendants kept their factual values. The descendant response discrepancy is the counterfactual part of Table 1. It compares the descendant responses of each edit with a reference, the image that the benchmark's generator renders from the intervened factors on ColoredBar-MNIST and a retrieved corpus image on the face benchmarks (Appendix E.5). Target success corresponds to the effectiveness criterion of Monteiro et al. (2023), and drift and the discrepancy follow from the graph and are defined in Appendix E.5.

Weakness 2, whether the metrics favor our method. Three features of the metrics could favor a method, and we checked each. First, on the face benchmarks one attribute predictor supplies both our selection measurements and the final scores. Under a separately trained CelebA classifier the target-success order is unchanged (comment above). Second, non-descendant drift rewards edits that change little, because an edit that barely changes the image also leaves the non-descendants fixed. We therefore read drift together with target success. Our lowest drift on StyleGAN3 comes together with the highest target success there, so it does not come from editing less. Third, the predictor may read low-fidelity images less reliably, so we report the Fréchet Inception distance next to every score. The metrics therefore do not by themselves favor our method, and the results show the same. BD-CLS has the lowest drift on Stable Diffusion XL and the lowest discrepancy on ColoredBar-MNIST, and ANCM the lowest drift on ColoredBar-MNIST and the lowest discrepancy on Stable Diffusion XL. CausalDiffAE (Komanduri et al., 2024) has the lowest Fréchet Inception distance on ColoredBar-MNIST, and the counterfactual generative network, built on the generator family of the StyleGAN3 benchmark, the lowest on StyleGAN3. Our StyleGAN3 path is deterministic given the factors, which lowers sample diversity and plausibly raises our distance there (comment above).

Taken together, our method is competitive on all three benchmarks without leading on every metric. In Table 1 it has the highest target success on all three, the lowest drift and discrepancy on StyleGAN3 and the lowest Fréchet Inception distance on Stable Diffusion XL. Two later results qualify this. When all five methods are trained in one ColoredBar-MNIST world, ANCM has a higher target success than ours (comment 3 to Reviewer H12i). On Stable Diffusion XL, one counterfactual generative network trained with the losses and learning rates of Sauer and Geiger (2021) has a higher target success and a lower drift than ours (Appendix E). The ColoredBar-MNIST rows of Table 1 were not trained in one data world (comment 1 to Reviewer H12i).

AuthorsOpenReview ↗

References for the comments above

References

Bińkowski, M., Sutherland, D. J., Arbel, M., and Gretton, A. (2018). Demystifying MMD GANs. International Conference on Learning Representations.

Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. (2012). A kernel two-sample test. Journal of Machine Learning Research, 13.

Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems.

Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. (2021). Alias-free generative adversarial networks. Advances in Neural Information Processing Systems.

Komanduri, A., Zhao, C., Chen, F., and Wu, X. (2024). Causal diffusion autoencoders: Toward counterfactual generation via diffusion probabilistic models. European Conference on Artificial Intelligence.

Liu, Z., Luo, P., Wang, X., and Tang, X. (2015). Deep learning face attributes in the wild. International Conference on Computer Vision.

Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. (2022). SDEdit: Guided image synthesis and editing with stochastic differential equations. International Conference on Learning Representations.

Monteiro, M., Ribeiro, F. D. S., Pawlowski, N., Castro, D. C., and Glocker, B. (2023). Measuring axiomatic soundness of counterfactual image models. International Conference on Learning Representations.

Pan, Y. and Bareinboim, E. (2024). Counterfactual image editing. International Conference on Machine Learning.

Pan, Y. and Bareinboim, E. (2025). Counterfactual image editing with disentangled causal latent space. Advances in Neural Information Processing Systems.

Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. (2024). SDXL: Improving latent diffusion models for high-resolution image synthesis. International Conference on Learning Representations.

Sauer, A. and Geiger, A. (2021). Counterfactual generative networks. International Conference on Learning Representations.

Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., and Cohen-Or, D. (2021). Designing an encoder for StyleGAN image manipulation. ACM Transactions on Graphics, 40(4).

Reviewer_T2ymOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to use a pretrained high-fidelity generative model, called an anchor, for counterfactual image editing within a formal causal framework based on structural causal models. One main bottleneck is that multiple editing models can match the observational distribution while implying substantially different counterfactual distributions.

To address this problem, the paper formalizes stable disagreement, which measures changes to non-descendant attributes relative to the factual image, and correction disagreement, which measures differences from the anchor's responses. It proposes a training-and-selection framework (MSMS) that trains candidate neural causal models and selects a model from the candidate pool by minimizing correction disagreement subject to an observational-fitting and a stable-disagreement constraint (Eq. 4, Algorithm 1). The paper provides a finite-query bound on how closely the selected model agrees with the anchor relative to the best candidate in the pool (Thm. 1), and a counterfactual-risk guarantee that holds under the condition that excess correction disagreement bounds the counterfactual risk (Thm. 2). Experiments on conducted on ColoredBar-MNIST and synthetic-face benchmarks using SDXL and StyleGAN backbone. The causal metrics (target success, non-descendant drift, descendant response discrepancy) and image quality (TID, FID) are report.

Critical strengths

S1. The paper formalizes comparisons between pretrained generators, which need not be causal models, and causal models through a shared query-response interface (Def. 3).

S2. The theoretical framework and proofs are clearly presented, with the guarantees explicitly conditioned on stated assumptions.

S3. The studied problem is interesting and important.

Critical weaknesses

Anchor editing pipeline

W1. The editing pipeline associated with the anchor is not clearly specified in the paper. The main text describes the anchor only as a pretrained generator that "edits a source image toward requested attributes". A pretrained generator such as SDXL or StyleGAN does not, by itself, define how a source image should be edited. It is itself a text-to-image generation model not a image model; The actual procedures appear only in App. E: SDXL img2img; StyleGAN3 “first finds a latent code in W+ that reconstructs the source, then changes it along attribute directions.”. CBMNIST img2img. 

These procedures are central to the method and should be introduced explicitly when defining the anchor. Different editing procedures built around the same generator can induce different response distributions and therefore different editing results. The paper should distinguish the pretrained generator from its editing pipeline and clarify how a editing pipeline can influence the selection criterion and the theoretical assumption.

Mismatch between selection and final editing

W2. The MSMS implementation on the face benchmarks has a mismatch between candidate selection and final image generation. During training and selection, candidate NCM outputs are passed through a learned scoring renderer (an MLP followed by a convolutional decoder, App. E.1), and attribute predictions on those images determine the selection scores. During final editing, the selected NCM's attributes are instead rendered by the pretrained editing pipeline. Consequently, selection scores reflect the auxiliary renderer as well as the NCM, and good scores need not imply accurate final edits.

Questions for authors

Q1. The paper motivates the scoring renderer by differentiability, since it "keeps image-based alignment differentiable without differentiating through the anchor". This explains its use during training, but selection does not require gradients. Why not render candidate attributes with the final pretrained editing pipeline during selection?

Q2. How often does the model selected by MSMS differ from the last training checkpoint? Could you report Table 1 for the last checkpoint without selection, so that the contribution of the selection step can be separated from that of the training objective?

3 comments

Thank you for the careful reading and for recognizing the shared query-response interface (Strength 1) and the explicitly conditioned guarantees (Strength 2).

Weakness 1, the editing pipeline. We agree, and the revision defines the anchor as a pretrained generator together with its editing procedure where the anchor is introduced (Section 3.2), with a table of the three procedures in the appendix. These are image-to-image editing in the manner of stochastic differential editing (Meng et al., 2022) with Stable Diffusion XL (Podell et al., 2024) at strength 0.70 and guidance scale 7.5, StyleGAN3 (Karras et al., 2021) with encoder-based inversion (Tov et al., 2021) into the extended latent space W+ (Abdal et al., 2019) followed by attribute-direction edits, and, for the final ColoredBar-MNIST (Pan and Bareinboim, 2024, 2025) edits, the same image-to-image editing with a class-conditional denoising diffusion model (Ho et al., 2020) at strength 0.85. Definition 3 uses the anchor only through its response kernel, so the selection criterion and the condition of Theorem 2 refer to the kernel of this pair. A different procedure defines a different kernel, which changes the correction score and can change whether the condition holds. Our data show this effect, since on Stable Diffusion XL the descendant coupling that the anchor prefers changes with the editing strength (0.50, 0.70 and 0.85 gave monotone, independent and antitone, comment 2 to Reviewer H12i). The revision states that rankings and guarantees are relative to the declared procedure and that architecture independence refers to the common query interface.

Weakness 2 and Question 1, selection with the final pipeline. Selection needs no gradients, so we ran it with the final pipeline. On Stable Diffusion XL we rescored all saved checkpoints that remained from the three runs (9, 6 and 5, including the selected and last ones) with the same selection bank, queries, 32 responses per query and threshold policy. On StyleGAN3 we rescored all 30 iterates. Final-pipeline selection chose earlier iterates in all three Stable Diffusion XL runs and in two of three StyleGAN3 runs, and these have a higher descendant response discrepancy on the final bank (Stable Diffusion XL 0.406 against 0.375, 0.378 against 0.363 and 0.391 against 0.371, mean difference 0.022 [0.004, 0.043] over sources, StyleGAN3 0.351 against 0.253 and 0.346 against 0.268). We read this through Proposition 4 (Appendix D.12), which splits Equation (6) into a query condition and an anchor condition, the latter requiring the anchor's own edits to lie close to the true counterfactual responses. The final pipeline's reference edit realizes the requested target only partially in some cells, for example male-to-female edits whose target is read with mean probability 0.11 to 0.21, and a distance to that reference favors candidates that share the shortfall. On Stable Diffusion XL the correction scores computed with the two renderers rank the candidates in reverse (Spearman −0.97, −1.00 and −1.00). In an earlier rescoring with eight responses per query, in which the final renderer also held the non-descendants at their factual values, dropping the queries whose reference misses the target, by a rule we wrote after seeing the reversal and fixed before running it, reduced the reversal in two runs (from −1.00 and −0.94 to +0.08 and −0.14) and not in the third (from −0.90 to −0.97). Final-pipeline selection therefore requires an anchor that realizes the requested target and a target reading that does not respond to the descendants, and the revision states both preconditions. The revision adds both rescorings to the appendix and describes Figure 4 as an association on the StyleGAN3 pools.

Question 2, the last checkpoint. Selection returned the last checkpoint in 8 of the 9 reported pools, so the last-checkpoint Table 1 equals Table 1 on ColoredBar-MNIST and StyleGAN3. On Stable Diffusion XL one run selected step 198 of 200, and the last-checkpoint row reads target success 0.661, non-descendant drift 0.075 and descendant response discrepancy 0.369, against 0.660, 0.075 and 0.370. The differences in Table 1 are therefore those of the editing pipeline trained with Equation (7), and selection is the rule through which Theorems 1 and 2 apply to the returned model. The revision adds this row, attributes the advantage over BD-CLS (Pan and Bareinboim, 2025) in Section 5.2 (line 423) to the trained pipeline, and states the executed selection rule in Section 3.6 and Algorithm 1, namely the stable threshold applied to all saved candidates and the correction score ranking those that pass. The submitted Section 3.6, Algorithm 1 and Appendix E.2 also describe an observational-fit screen (ε_obs = 0.95) that the runs did not apply, and the revision states in Section 3.6 and Appendix E.2 that the runs did not apply it.

AuthorsOpenReview ↗

References for the comment above

References

Abdal, R., Qin, Y., and Wonka, P. (2019). Image2StyleGAN: How to embed images into the StyleGAN latent space? International Conference on Computer Vision.

Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems.

Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. (2021). Alias-free generative adversarial networks. Advances in Neural Information Processing Systems.

Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. (2022). SDEdit: Guided image synthesis and editing with stochastic differential equations. International Conference on Learning Representations.

Pan, Y. and Bareinboim, E. (2024). Counterfactual image editing. International Conference on Machine Learning.

Pan, Y. and Bareinboim, E. (2025). Counterfactual image editing with disentangled causal latent space. Advances in Neural Information Processing Systems.

Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. (2024). SDXL: Improving latent diffusion models for high-resolution image synthesis. International Conference on Learning Representations.

Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., and Cohen-Or, D. (2021). Designing an encoder for StyleGAN image manipulation. ACM Transactions on Graphics, 40(4).

W1. My concern is addressed. The new explanation about anchor (line 199) is more clear now.

W2, Q1 and Q2. My concern is partially resolved. Thanks for added the experiments about final render selection. The added experiments show that the descendant response discrepancy is higher when using render selection. And the explanation in the exps illustrate the potential flaws. However, the two renderers rank candidates in nearly opposite orders on SDXL, so it is still not clear why a learned renderer that differs so much from the final pipeline helps the final edits.

In addition, the answer to Q2 also shows that selection returns the last checkpoint in most runs. This limit the value of selection algorithm since the most common choice is to chose the last checkpoints. The value of selection should be presented in a better way even the theorem is sound.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.