acceptodds
Under review as a conference paper at ICLR 2027

Latent Structural Causal Modeling for High-Dimensional Counterfactual Generation

Abstract

High-dimensional counterfactual generation must bridge the dimensional gap between causal variables and complex observations while preserving unit-specific variation. We introduce a manifold-guided factorization that maps causal nodes into a shared low-dimensional latent space. An additive structural causal model performs abduction, intervention, and prediction while preserving semantic exogenous residuals, and a conditional flow-matching generator models observation-level noise and renders the terminal state. On Morpho-MNIST, our method recovers causal mechanisms and produces graph-consistent counterfactuals; on OASIS brain MRI, it generates subject-preserving, anatomically ordered images under demographic, anatomical, and dementia-severity interventions within a working graph. Together, these results establish low-dimensional structural propagation as a practical approach to counterfactual generation for high-dimensional observations.

Primary area:
causal reasoning
Keywords:
Structural Causal Models, Counterfactual Generation, Conditional Flow Matching, Causal Representation Learning, Medical Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_F3DGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes separating high-dimensional counterfactual generation into an additive latent SCM and a conditional image generator. Counterfactual inference preserves abducted latent residuals, propagates interventions through the latent graph and estimates the final state using conditional flow matching. The paper provides conditional identification and generalisation results with experiments on Morpho-MNIST and OASIS MRI.

Critical strengths
  • The separation between causal estimation and image synthesis provides a clear modular framework. Explicitly distinguishing structural residuals from rendering noise helps clarify which variation should persist across counterfactual worlds. Moreover, the paper carefully acknowledges its assumptions and limitations, including representation validity, overlap, the model dependence of fixed-noise pairing, and the illustrative status of the OASIS causal graph.

  • On the empirical front, the known-SCM experiment includes a useful evaluation before image generation. For example, in Table 2, preserving intensity residuals improves counterfactual intensity NRMSE from 0.384 to 0.114, indicating strong performance gains. The ablations also demonstrate the importance of structural propagation and residual preservation.

Critical weaknesses
  • An existing paper (Peng et al., Latent Causal Modeling for 3D Brain MRI Counterfactuals) already encodes images, fits a latent structural mechanism, abducts and preserves its residual, propagates counterfactual parents, and decodes the resulting latent into an image. This substantially overlaps with the proposed separation of causal propagation from image synthesis. Although this paper (the one under review) adds some additional flexibility, it does not clearly establish what differences constitute the main advance nor does it demonstrate its benefit through a controlled comparison. This is particularly important because the factorisation itself is presented as the central contribution.

  • Another, more specific, weakness is that the experimental evidence is insufficient to establish the benefit of the proposed separation between latent causal propagation and image generation. On Morpho-MNIST, the residual-preservation experiment demonstrates the importance of retaining unit-specific information, but does not establish an advantage over existing residual-preserving latent SCMs. Although the paper includes a common-bottleneck ablation, its replacement architecture is insufficiently specified to isolate the contribution of the factorisation from differences in model capacity or optimisation. A controlled comparison using the same image generator and comparable structural-model capacity would strengthen this claim. In addition, the OASIS experiments lack quantitative baseline comparisons and measurements of subject preservation, limiting evidence of practical improvement. All analyses could be improved with the inclusion of uncertainty estimates across independent training runs, which would make the robustness of the reported gains easier to assess.

Questions for authors
  • What is the methodological advance over Peng et al.’s latent residual-preserving SCM followed by image decoding?
  • Which experiments resample rendering noise, reuse a sampled seed, or reuse factually inverted noise?
  • Which counterfactual estimand does each protocol evaluate?
  • How are nonterminal encoders constrained, and how are predicted latent states converted back into scalar attributes for Table 2?
  • Are OASIS experiments split by subject across all slices and visits, and what precisely replaces the common bottleneck in the ablation?
1 comment

Response to Reviewer 3

W1/Q1 (Peng et al.). We will cite and discuss Peng et al. in the main text. It is the closest prior instance of "encode, fit a latent mechanism, preserve the residual, propagate, decode", and we do not claim that pipeline is new. The differences:

  1. Latent nodes. Peng et al. embed only the image; attributes are modeled by a DSCM, and the flattened image code (d×h×w×nDd\times h\times w\times n_D) remains a high-dimensional child of a few scalars. We embed every node in a shared mm-dimensional space, which targets the parent–terminal dimensional gap and lets Theorem 2 scale with mm.
  2. Residual vs. observation noise. With a deterministic decoder, their residual UZ=Z−PBU_Z=Z-PB mixes unit-specific variation with observation-level detail. We separate Uk\bm U_k from the generator noise ϵ\bm\epsilon, giving two well-defined couplings (Sec. 3.3) and an identified counterfactual law (Theorem 1).
  3. Representation. Their VQ-VAE is trained for reconstruction alone; our terminal encoder is trained jointly with a flow-matching generator of P(X∣hK)P(\bm X\mid\bm h_K), which absorbs image-level variability.
  4. Mechanisms. OLS-fit linear versus learned nonlinear additive mechanisms.
  5. Theory. We add identification (Thm. 1), an error decomposition (Prop. 1) and a generalization bound (Thm. 2).
  6. Evaluation. They assess 3D realism; we evaluate known-SCM recovery and soundness (no 3D claim).

Controlled comparison. Peng et al.'s structural component is a residual-preserving linear latent SCM. Keeping our frozen encoder and generator and refitting only a linear SCM, the Morpho-MNIST effectiveness MAE worsens: do(thickness) 0.23→0.32, do(intensity) 6.4→9.3, do(slant) 3.0→5.2, and the propagated intensity after do(thickness) 6.0→10.1. Reconstruction (0.0086) and reversibility are unchanged, so the gain comes from the structural mechanism. On OASIS, fresh ϵ\bm\epsilon changes an image by only 5.6×10−45.6\times10^{-4} (brain-masked L1L_1); counterfactual content is carried by hK\bm h_K.

W2 (evidence). The linear variant and the inference-time ablations (no propagation, no residual) share the encoder and generator, isolating the structural model; the linear variant is itself a residual-preserving latent SCM, complementing Table 2 (RMSE 18.558→5.561). On OASIS, the null intervention leaves hK\bm h_K unchanged (≤1.3×10−3\le1.3\times10^{-3}), removing the residual lowers subject re-identification of held-out reconstructions from 0.82 to 0.41, and noise-floor and spectral analyses (R1 W2) rule out generation variability. All results are single-run, which we will state.

Q2 (noise). All paper experiments resample ϵ\bm\epsilon freshly ("Noise type in inference", appendix); none reuse a seed or inverted noise, which we will state in the captions. Morpho-MNIST L1L_1 for fixed / fresh / inverted noise: reconstruction 0.0086/0.0085/0.0085; reversibility do(thickness) 0.032/0.031/0.034, do(intensity) 0.012/0.011/0.017, do(slant) 0.013/0.012/0.018, do(digit) 0.027/0.027/0.032. Conclusions do not depend on the convention and lie above the fresh-noise floor (0.0059); on OASIS the three agree to three decimals.

Q3 (estimands). All protocols evaluate, via samples, the fresh-noise counterfactual law P(Xa∣h=hobs)P(X^a\mid\bm h=\bm h^{\mathrm{obs}}) of Theorem 1. Effectiveness compares generated attributes with targets; reversibility and null intervention are consistency checks, not paired individual estimates; Table 2 evaluates the latent descendant before generation. The fixed-noise pairing is not identified (Sec. 3.4) and serves only as a robustness check. We will add an estimand table.

Q4 (encoders, Table 2). Each non-terminal node uses one linear layer into the latent space; EKE_K is trained with flow matching and frozen, anchoring the scale, and the rest is fit with LSCM\mathcal L_{\mathrm{SCM}}. Latents do not collapse: the effective rank of hK\bm h_K is 30/128 on Morpho-MNIST (22 for a random encoder) and 178/384 on OASIS (50), with no dead dimensions. Linear probes recover the attributes from mK(pa)m_K(\mathrm{pa}) (digit acc. 0.999; R2≥0.98R^2\ge0.98) but hardly from UK\bm U_K (0.28; R2≤0.26R^2\le0.26). For Table 2, PCA is fit jointly on predicted and true counterfactual latents and both are projected onto PC1; RMSE is in PC1 units, the same affine map applies to both, so NRMSE is the interpretable number.

Q5 (split, ablation). The OASIS split is by session, stratified by clinical group: 299 sessions (6,279 slices) for training, 74 (1,554) for evaluation; all slices of a session stay together, but other visits of an evaluation subject may be in training, so we will call these held-out sessions, not unseen subjects. Ablation: the 128-d CLS state hK\bm h_K is replaced by 49 patch tokens (7×77\times7, 128-d each) predicted from the parents by a shared terminal decoder, giving 6272 terminal coordinates and reintroducing the dimensional gap; parameter counts match. We will specify this in the appendix.

Reviewer_sGQmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work presents a method for high-dimensional counterfactual generative modelling that separates causal calculus in a semantic latent space from low-level rendering variability. The image and covariates are encoded into low-dimensional representations, whereupon latent SCM mechanisms (with additive noise) capture their causal relationships. The image is rendered via a conditional flow matching model, and counterfactual inference follows Pearl's abduction–action–prediction paradigm. The authors provide some theoretical results on identifiability and generalisation. The method is evaluated on synthetic Morpho-MNIST data following a protocol from prior work, and on 2D MRI slices from the OASIS dataset.

Critical strengths

The methodology is sensible, relatively simple, and generally well motivated. It builds on a growing body of work leveraging causality principles and SOTA generative modelling to enable sound counterfactual inference even on high-dimensional data such as images. The authors additionally attempt to provide certain theoretical guarantees on identifiability and generalisation (which I don't have the right background to fully verify).

Critical weaknesses

One of my main concerns is the unclear positioning of this work relative to prior art. The authors cite many related works on deep counterfactual modelling and claim theirs differs along four axes overall. However, they do not describe the limitations of specific prior methods nor how the proposed method addresses them (even in the extended discussion in Appendix D), making it hard to judge the novelty and significance of the contributions. As I detail below, the experimental results also do not clearly support claims of improvement over the evaluated external baselines. Lastly, the authors do not engage with prior work on causal and counterfactual identifiability when presenting their own theoretical results.

Another core limitation of the study is weak empirical evidence. While the synthetic Morpho-MNIST toy setting is valuable for validating the method against a known ground-truth SCM, it is not sufficient by itself to corroborate real-world applicability. Also, and related to my earlier point, the evaluated baselines are not described in enough detail to communicate how they differ from the proposed method or whether they are fair comparisons. Further, all baseline results seem copied from Rasal et al. (2025) rather than independently reproduced (which should be clearly noted in the text), raising concerns about direct comparability. As the only "real-world" setting, the MRI experiment reports mainly qualitative results (plus very limited quantitative results in the appendix only), and includes no baselines or ablations for reference.

Questions for authors
  1. Elaborate the related work discussion to better position your paper: Which counterfactual methods are similar or different according to your four criteria? Which exact limitations are addressed by your claimed contributions? How do your identifiability and generalisation derivations relate to analogous theory developed in other works?
  2. Justify the suitability of the external baselines: Why were those specific baselines selected? Are they the closest and most informative comparisons, or simply the ones already evaluated in prior work? What are their architectural details, and are they a fair match with yours? Assuming the results were indeed transferred directly from prior work, why couldn't they be reproduced/reimplemented to ensure head-to-head comparison?
  3. Expand quantitative analyses on MRI case study: Though effectiveness can be hard to estimate in this case (because it requires accurate covariate predictors), other properties like reversibility and null-composition could readily be measured as sanity checks. Also, could you add any baselines or ablations (even if inference-time only), to put the results into perspective?
2 comments

Response to Reviewer 2

Thank you for the detailed feedback.

W1/Q1 (positioning). We will add a per-method table to App. D. In short:

  • DSCM (Pawlowski et al., 2020): flow mechanisms among attributes; the image is a conditional-VAE child of the attributes; its noise is split into a high-level zz and pixel noise uu, both abducted and reused at the image level.
  • HVAE (Ribeiro et al., 2023): as DSCM, with a hierarchical decoder conditioned on the parents at every layer; z1:Lz_{1:L} is a mediator inside the image mechanism, not a state shared with other nodes.
  • VCI (Wu et al., 2025): no attribute graph; treatments and one abducted latent feed an encoder–decoder; outcome noise is set to zero, so residual and observation noise are not separated.
  • DiffSA (Rasal et al., 2025): attribute mechanisms plus a conditional diffusion image mechanism; the semantic code zz and DDIM-inverted uu are both noise of the image mechanism.
  • Peng et al. (2025): DSCM over attributes; an image code linear in the parents whose residual is reused; deterministic decoder, hence no observation noise.

In all of them the image mechanism is learned inside the image model, mapping a few low-dimensional parents directly to a high-dimensional observation. We address (i) the dimensional gap: every mechanism, including the one producing hK\bm h_K, operates among comparably low-dimensional latent states, and only synthesis is left to the generator; (ii) the role of noise: UK\bm U_K is a node-level residual of the latent SCM, whereas ϵ\bm\epsilon is the observation noise of a separate generator P(X∣hK)P(\bm X\mid\bm h_K), which yields the two couplings of Sec. 3.3 and Theorem 1. Identifiability. Theorem 1 identifies the counterfactual law P(Xa∣h=hobs)P(X^a\mid\bm h=\bm h^{\mathrm{obs}}) conditional on a consistent representation, an additive latent SCM, regime invariance and overlap; it does not claim representation recovery. We will relate it to classical counterfactual identification (Shpitser & Pearl, 2006), counterfactual identifiability of deep SCMs with bijective mechanisms (Nasr-Esfahany et al., 2023), and identifiable/causal representation learning (Khemakhem et al., 2020; Schölkopf et al., 2021), which addresses the representation condition we assume. We identify a distribution, not a unique pairing, consistent with the non-identifiability of individual counterfactuals. Theorem 2 is an Occam-type bound (Blumer et al., 1987); our point is the scaling with the latent dimension mm rather than the ambient DD.

W2/Q2 (baselines). We follow Rasal et al. (2025) (Sec. 4) and compare with the methods they evaluate (VAE, HVAE, VCI, DiffSA) on the same Morpho-MNIST SCM and soundness metrics, the established benchmark for this task. We confirm that the baseline numbers in Table 1 and the appendix soundness table are taken from Rasal et al. and will state this in the text and captions. We did not re-run them because reimplementing four methods risks under-tuned baselines and protocol drift; we will note this limitation. Baselines use their source architectures, and the comparison is protocol-matched. Our model sizes: Morpho-MNIST flow 2.29M, SCM 1.59M; OASIS DiT-B (patch 6; encoder 43.0M, decoder 29.1M), SCM 33.1M. As a controlled comparison sharing our encoder and generator, a linear structural mechanism degrades effectiveness for every intervention (MAE: thickness 0.32 vs. 0.23, intensity 9.3 vs. 6.4, slant 5.2 vs. 3.0) with unchanged reconstruction and reversibility (see also R1 Q3).

Q3 (MRI quantitative analyses). On 74 held-out sessions (58 subjects) not seen by the generator or the SCM:

  1. Null intervention: hK\bm h_K is unchanged (max deviation ≤1.3×10−3\le1.3\times10^{-3} per node), so it equals the reconstruction under every noise convention.
  2. Noise floor: fresh-ϵ\bm\epsilon regeneration gives a brain-masked L1L_1 of 5.6×10−45.6\times10^{-4} and a ventricular-fraction SD of 8×10−48\times10^{-4}; fresh-, fixed- and inverted-noise counterfactuals agree to three decimals.
  3. Patch artifacts: no pronounced spectral peak at the DiT patch frequency (R1 W2).
  4. Inference-time ablation: removing the abducted residual lowers subject re-identification from 0.82 to 0.41.
  5. Latent diagnostics: hK\bm h_K has effective rank 178/384 (50 for a random encoder), and swapping hK\bm h_K changes the image ~180× more than resampling ϵ\bm\epsilon.

All results come from a single training run, which we will state, and we will align the OASIS claims with its role as a showcase.

Q1: Thank you for contextualising the theory references and for detailing how the prior methods fit along your four criteria. Note that DSCM (hence also HVAE and Peng et al.) and DiffSA do indeed abduct unit-specific factual information for every node, which is reinjected for non-intervened nodes at prediction time. The phrasing in Appendix D seems to imply otherwise, which is misleading. Also, the added discussion of limitations of related methods still does not explain why the dimensional gap might be a problem, nor why the proposed noise factorisation is better than the alternatives.

Q2: Unfortunately, without any architectural details of the external baselines in the paper, readers cannot tell whether the reported improvements are due to the proposed methodology or are confounded by differences in training objective, model architecture/size, training data, etc. Even without full reproduction, such details must be provided to contextualise the results. The new linear-mechanism baseline is a strong addition, but does not fully address my concern by itself.

Q3: The new OASIS quantitative analyses are valuable sanity checks, and the authors have clarified the role of the experiment more appropriately as a demonstration rather than a systematic validation. That said, the study still lacks other benchmarks with baselines to bolster the empirical evidence.

Reviewer_vvhzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a latent structural causal modeling framework for high-dimensional counterfactual image generation. The method separates causal propagation from image synthesis by performing interventions in a low-dimensional latent SCM and then using a conditional generator to render the resulting image. The framework is evaluated on Morpho-MNIST, where the true causal structure is known, and on OASIS brain MRI as a real-world demonstration using a working causal graph.

Critical strengths
  1. The separation of low-dimensional causal propagation from high-dimensional image synthesis is conceptually clean and well motivated.
  2. Morpho-MNIST provides a known SCM, allowing direct evaluation of structural recovery, intervention propagation, and preservation of unit-specific variation.
  3. The paper provides explicit identification assumptions, an error decomposition, and generalization results rather than presenting only an empirical generative model.
Critical weaknesses
  1. The real-world validation is limited. In OASIS, there is no ground-truth individual counterfactual MRI, and the evaluation mainly demonstrates anatomically plausible changes rather than causal correctness.

  2. The OASIS counterfactuals show visible patch/stitching artifacts and substantial local changes in cortical anatomy, raising concern that the difference maps may reflect generation artifacts rather than meaningful intervention effects.

  3. The working causal graph is relatively sparse and may omit important common causes of dementia severity and brain structure, making the counterfactuals potentially sensitive to unmeasured confounding and graph misspecification.

Questions for authors
  1. Since OASIS is longitudinal, could the authors use later scans from the same subjects as a partial validation target, for example by testing whether counterfactuals generated from earlier visits move toward the observed future anatomy?

  2. For the sex intervention in Fig. 3, are the localized differences biologically meaningful and consistent with known sex-related neuroanatomical patterns, or could they reflect generator artifacts? Why was the local cortical anatomy changed substantially, including the shape of gyri and sulci?

  3. Can the authors more directly isolate whether the performance gains come from the proposed structural-generative factorization itself rather than differences in the encoder, generator, or training setup?

1 comment

Response to Reviewer 1

Thank you for the constructive review. All OASIS numbers below come from our improved OASIS-2 pipeline (150 subjects, 373 sessions, MNI152 2 mm, 84×8484\times 84 axial slices; generator and latent SCM fit on training sessions only), and the paper's OASIS figures and tables will be updated accordingly.

W1/Q1 (real-world validation, later scans). We agree that OASIS has no ground-truth counterfactual and supports plausibility and subject preservation, not causal correctness; this is why soundness is assessed on Morpho-MNIST (known SCM) and OASIS is framed as a working-graph showcase. We ran the suggested longitudinal test on 372 ordered visit pairs (58 subjects; 198 forward pairs, mean gap 2.7 y; registered slices 40/45/50): residuals abducted at the source visit, intervention with the target-visit age (propagated) or all target covariates, compared with a no-change reference, a no-propagation counterfactual and a no-residual model. The test is uninformative at the slice level: the mean change in ventricular fraction over follow-up is 0.003, below its visit-to-visit variability (mean ∣Δ∣|\Delta| 0.013), so slice-level change is dominated by acquisition and registration factors outside the graph, and our counterfactuals are not closer to the later scan than the no-change reference. We will report this as a limitation; volumetric data or longer follow-up would be needed, and even then agreement with progression would not identify a single covariate's effect (W3). Ground-truth-free checks: (i) the null intervention leaves hK\bm h_K unchanged (max deviation ≤1.3×10−3\le1.3\times10^{-3} for every node); (ii) removing the abducted residual lowers subject re-identification of held-out reconstructions from 0.82 to 0.41; (iii) noise-floor and spectral analyses (W2).

W2/Q2 (artifacts, cortical changes, sex). Patch artifacts. The DiT uses patch size 6 on 84×8484\times84, so tokenization artifacts would peak at 14 and 28 cycles/image. Spectral prominence of difference maps (1 = no peak):

k=14k=14k=28k=28k=17k=17 (control)
Real image0.900.920.94
Generated image0.910.970.93
CF, age +10+101.101.080.95
CF, CDR flip1.151.090.95

All values stay close to 1, with no pronounced periodic peak (Morpho-MNIST, k=7k=7: 0.80 real, 0.84 counterfactual). Observation noise. ϵ\bm\epsilon is the generator's observation-level noise, present in every acquired image and separate from the structural residuals Uk\bm U_k. Regenerating factual images with fresh ϵ\bm\epsilon gives a brain-masked L1L_1 of 5.6×10−45.6\times10^{-4} and an SD of 8×10−48\times10^{-4} in ventricular fraction, and fresh-, fixed- and inverted-noise counterfactuals agree to three decimals. The factual–counterfactual differences therefore come from the change in hK\bm h_K propagated through the latent SCM, as intended. We will add a noise-floor map to Fig. 3 and present the anatomical changes as qualitative illustrations under the working graph, not as clinically validated effects. Sex. We do not claim biological validity and will remove the suggestive sentence in Sec. 4.2. In the graph, sex acts directly and through nWBM and CDR, so its counterfactual mixes direct and mediated effects that the graph cannot localize; support is also uneven (59% female).

W3 (sparse graph). Agreed. The graph (age, sex → nWBM, CDR, MRI; education → CDR; nWBM, CDR → MRI) uses only the available variables and is stated not to be identified or clinical (Sec. 4.2, Ethics Statement). Identification requires the latent DAG to include the relevant common causes (Assumption 1), and App. D notes that violations can mislead. We will make this more prominent and describe the OASIS counterfactuals as conditional on the stated graph.

Q3 (isolating the factorization). On Morpho-MNIST we hold the encoder, the generator and its training fixed and change only the structural model:

  • Linear mechanism (latent SCM refit): effectiveness MAE is worse for every intervention (thickness 0.32 vs. 0.23, intensity 9.3 vs. 6.4, slant 5.2 vs. 3.0; propagated intensity after do(thickness) 10.1 vs. 6.0), while reconstruction (0.0086) and reversibility are unchanged.
  • No propagation / no residual (inference-time, same model): residual preservation reduces the latent-descendant RMSE from 18.558 to 5.561 (Table 2).
  • No common bottleneck: the 128-d CLS state hK\bm h_K is replaced by 49 patch tokens (7×77\times7, 128-d each) predicted from the parents by a shared terminal decoder, i.e. 6272 terminal coordinates, which reintroduces the dimensional gap; parameter counts are matched. We will specify this in the appendix.

Comments on the paper

1 comment
Area_Chair_4qxeOpenReview ↗

Encouraging Reviewer Feedback on Author Responses

Dear Reviewers,

Thank you all for your time and effort in reviewing and responding. Just a gentle reminder to reviewers to take a look at the authors' responses and share any updated thoughts on whether your main concerns have been addressed and, if not, to indicate which points are not well addressed. The author–reviewer discussion period ends on November 18 (AoE), so I would greatly appreciate your input before then. Thank you again for your time and contributions!

Kind Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.