Beyond Feasibility: Minimal-Shift Proxy Selection for Counterfactual Image Editing

Counterfactual image editing asks how a given image would change under an intervention on a semantic attribute, accounting for causal effects while preserving unaffected attributes. Observational data and a causal graph cannot generally determine this change uniquely, as observationally equivalent models may imply different counterfactual images. Existing methods characterize admissible solutions within partial identification bounds but leave their selection unresolved, and their implementations can require full generative training or image–factor invertibility. To address these limitations, we propose Minimal-Shift Mechanism Selection (MSMS), which uses a large pretrained generator as additional evidence for ranking counterfactually consistent mechanisms. MSMS favors candidates whose editing responses most closely match those of the pretrained model, subject to preserving unaffected attributes. We formulate this comparison in a shared space across heterogeneous architectures and establish sufficient conditions for $L_3$-consistency under realizability and reference calibration. By separating factor abduction from image rendering, MSMS reuses pretrained rendering capabilities without requiring an invertible mapping between factors and images, supporting diffusion models and GANs. Experiments across three benchmarks demonstrate MSMS's ability to perform targeted edits while preserving non-descendant attributes. We further examine how pretrained responses guide the selection of candidate causal models.