acceptodds
Under review as a conference paper at ICLR 2027

Generating Chest X-Ray Counterfactuals by Specialising Foundation Image Models

Abstract

Counterfactual image generation answers questions about how a subject would have looked under retrospective, hypothetical scenarios. Recent methods have improved perceptual quality, identity preservation and faithfulness to an underlying causal model, but their adoption in healthcare is limited by scarce annotated data, distribution shift between datasets, and mismatches between pretrained generative models and those required for counterfactual inference. We propose specialisation, a data and parameter-efficient framework for adapting pretrained, non-causal generative models into causal mechanisms under distribution shift. Based on this framework, we train a radiology counterfactual image generation model, called RadCF, using latent flow matching. We validate our approach on three chest X-ray datasets spanning different dataset shifts, data volumes, and counterfactual questions, associated with challenging, highly-localised interventions. Our results show that RadCF and specialisation improves counterfactual soundness over existing methods while being data and parameter efficient, and that the resulting counterfactuals can detect and mitigate shortcut learning in a downstream medical classifier. Code available at https://anonymous.4open.science/r/RadCF-D7B5/.

Primary area:
causal reasoning
Keywords:
Counterfactual Image Generation, Medical Image Analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LEFCOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces specialisation, a framework for adapting pretrained image generators into counterfactual image mechanisms using limited target-domain data. They introduce RadCF, which implements this framework through latent flow matching, structured metadata conditioning, and adaptation of both the transport model and autoencoder. Experiments examine acquisition shift, data scarcity, extension to new intervention variables, adaptation of text-conditioned generators, and downstream shortcut detection and mitigation.

Critical strengths
  • The paper addresses a very practically important problem, that counterfactual image models often require substantial annotated data and retraining for each domain. Reusing pretrained generators offers a useful route toward more accessible counterfactual modeling.

  • The evaluation is extensive and includes several valuable controls. These include comparisons using the same RadEdit backbone, six independent effusion evaluators, patient-level data separation, and downstream experiments with size-matched augmentation strategies. For example, the acquisition-shift experiments provide convincing evidence for adapting the autoencoder.

Critical weaknesses
  • The reported soundness metrics provide limited evidence of clinically faithful counterfactual edits. Effectiveness is measured using pseudo-oracle classifiers, while composition and reversibility primarily assess reconstruction and cycle consistency, and these criteria do not establish that an intervention preserves clinically relevant findings outside the four modelled attributes. A focused assessment of target attainment and preservation of unrelated pathology would strengthen the contribution.

  • The methodological advance beyond established adaptation techniques is insufficiently isolated. Section 3.2 combines LoRA, joint autoencoder–generator adaptation, and replacement of text conditioning with structured metadata. The distinction among structural, acquisition, and measurement specialisation is useful, but the technical contribution consists mostly of selecting and combining these existing adaptation strategies. For example, table 2 establishes the benefit of autoencoder adaptation and table 3 demonstrates improvements over Causal-Adapter on the same backbone but I do not believe these results do not fully explain which particular design choice produces the advantage.

Questions for authors
  • How reliably do the interventions preserve clinically relevant findings outside the modelled attributes?
  • Which specific component of specialisation accounts for its advantage over existing adaptation approaches, and can this be isolated experimentally?
3 comments
AuthorsOpenReview ↗

Response to Reviewer LEFC (Part I)

We thank the reviewer for recognising the practical importance of our work and the value of the experimental controls. Below, we provide an additional assessment of unmodelled pathological findings and clarify the contribution of each specialisation component.

W1, Q1: The reported soundness metrics provide limited evidence of clinically faithful counterfactual edits.

W1: “The reported soundness metrics provide limited evidence of clinically faithful counterfactual edits. … these criteria do not establish that an intervention preserves clinically relevant findings outside the four modelled attributes. A focused assessment of target attainment and preservation of unrelated pathology would strengthen the contribution.”

Q1: “How reliably do the interventions preserve clinically relevant findings outside the modelled attributes?”

R1. We thank the reviewer for this suggestion. Upon request, we provide a focused assessment of non-target pathological preservation to further support the clinical faithfulness of our counterfactuals.

In this focused assessment, we examine whether predictions for unmodelled pathological findings remain consistent under view interventions. We focus on do(view) because changing between AP and PA views involves changes across the entire image, potentially affecting findings beyond the target attribute.

We compare predictions on original NIH-14 images and their view counterfactuals using two independent evaluators: RAD-DINO linear probes (Pérez-García et al., 2025b) and TorchXRayVision ResNet-50 (Version All, Cohen et al., 2022). For each finding, we report the percentage of original–counterfactual pairs whose predicted label changes. We include six findings that are not modelled by the SCM and for which the evaluators achieve an AUROC above 0.80 on factual images: pneumothorax, emphysema, mass, hernia, edema and cardiomegaly. Effusion, a modelled attribute held fixed during do(view), is included as a reference.

Despite the whole-image change required by the view intervention, predictions for the six unmodelled findings generally remain consistent among all models. And SA-SPEC follows closely behind the identifiable pixel-space flow. These results extend the assessment beyond reconstruction and cycle consistency to predictions of unmodelled pathological findings. We have added these results in Appendix B.4, complementing our existing evaluations.

Table R4. RAD-DINO: disease flip rate (%) under view intervention, with bootstrap 95% CI (10,000 resamples)

FindingAUCDiffusionImage-space flowRadCF (from scratch)RadCF s-specRadCF sa-spec
Pneumothorax0.898.18 [7.67, 8.68]6.96 [6.50, 7.42]3.93 [3.59, 4.29]8.25 [7.76, 8.74]6.50 [6.06, 6.95]
Emphysema0.862.33 [2.06, 2.61]1.59 [1.37, 1.81]1.45 [1.25, 1.67]1.45 [1.24, 1.66]1.65 [1.42, 1.88]
Effusion0.856.41 [5.96, 6.86]6.45 [6.01, 6.89]8.76 [8.26, 9.25]6.89 [6.44, 7.34]7.38 [6.91, 7.85]
Mass0.851.71 [1.48, 1.95]1.74 [1.52, 1.98]2.34 [2.07, 2.61]1.77 [1.54, 2.01]1.75 [1.52, 1.99]
Hernia0.850.91 [0.74, 1.08]0.71 [0.56, 0.86]0.13 [0.07, 0.20]1.74 [1.51, 1.98]0.40 [0.29, 0.52]
Edema0.850.38 [0.28, 0.50]0.41 [0.31, 0.53]0.93 [0.77, 1.11]0.26 [0.17, 0.35]0.47 [0.36, 0.61]
Cardiomegaly0.830.66 [0.51, 0.81]0.63 [0.49, 0.77]0.90 [0.74, 1.07]0.81 [0.65, 0.97]0.77 [0.62, 0.93]
Mean2.94 [2.82, 3.06]2.64 [2.53, 2.75]2.64 [2.53, 2.74]3.02 [2.91, 3.14]2.70 [2.59, 2.82]

Table R5. TorchXRayVision ResNet-50 (All): disease flip rate (%) under view intervention, with bootstrap 95% CI (10,000 resamples)

FindingAUCDiffusionImage-space flowRadCF (from scratch)RadCF s-specRadCF sa-spec
Hernia0.950.03 [0.00, 0.06]0.02 [0.00, 0.04]0.02 [0.00, 0.04]0.02 [0.00, 0.06]0.02 [0.00, 0.04]
Emphysema0.902.21 [1.94, 2.49]1.37 [1.17, 1.58]1.55 [1.34, 1.78]1.55 [1.34, 1.77]1.68 [1.47, 1.92]
Cardiomegaly0.880.04 [0.01, 0.09]0.05 [0.02, 0.09]0.07 [0.02, 0.12]0.06 [0.02, 0.11]0.07 [0.02, 0.12]
Edema0.870.02 [0.00, 0.04]0.02 [0.00, 0.04]0.02 [0.00, 0.06]0.00 [0.00, 0.00]0.02 [0.00, 0.04]
Effusion0.862.08 [1.81, 2.34]2.60 [2.32, 2.89]3.22 [2.91, 3.54]3.27 [2.96, 3.59]2.27 [2.01, 2.53]
Mass0.841.38 [1.17, 1.60]1.20 [1.01, 1.40]1.38 [1.17, 1.59]1.68 [1.45, 1.91]1.27 [1.07, 1.47]
Pneumothorax0.830.54 [0.42, 0.68]0.50 [0.38, 0.63]0.51 [0.39, 0.64]0.55 [0.42, 0.68]0.53 [0.41, 0.67]
Mean0.90 [0.83, 0.97]0.82 [0.76, 0.88]0.97 [0.90, 1.04]1.02 [0.95, 1.09]0.84 [0.77, 0.90]
AuthorsOpenReview ↗

Response to Reviewer LEFC (Part II)

W2, Q2: The methodological advance beyond established adaptation techniques is insufficiently isolated.

W2: “The methodological advance beyond established adaptation techniques is insufficiently isolated. Section 3.2 combines LoRA, joint autoencoder–generator adaptation, and replacement of text conditioning with structured metadata. The distinction among structural, acquisition, and measurement specialisation is useful, but the technical contribution consists mostly of selecting and combining these existing adaptation strategies.”

Q2: “Which specific component of specialisation accounts for its advantage over existing adaptation approaches, and can this be isolated experimentally?”

R2. Rather than relying on a single dominant module or heuristically combining techniques, our core contribution is a framework that maps three distinct clinical dataset shifts: structural, acquisition, and measurement to components, building on Castro et al., 2020. By identifying which component requires updating under a given shift, our framework achieves clear advantages in data and parameter efficiency over naive adaptation.

Importantly, we experimentally examine the contribution of these adaptation choices. Section 4.1 (When is SA-SPEC necessary?, Table 2) directly compares joint VAE+generator adaptation with generator-only adaptation using a frozen VAE. Under CheXpert-to-NIH-14 transfer, even full generator fine-tuning does not recover SA-SPEC’s composition and reversibility benefits. Section 4.2 (Causal Text-to-Image Generation, Table 3) evaluates measurement specialisation on a second backbone family. Using the same pretrained RadEdit backbone, M-SPEC’s direct structured conditioning and LoRA generator adaptation improve target control, composition, reversibility and realism over Causal-Adapter.

These controlled comparisons directly address an open gap in prior work. Earlier SCM-based methods primarily study learning and inference for dataset-specific mechanisms (Ribeiro et al., 2023; Rasal et al., 2025; Ribeiro et al., 2025). Meanwhile recent adaptation studies (Yeganeh et al., 2025; Tong et al., 2026) only study adapting general-purpose pretrained models for medical or counterfactual generation. Our study explicitly tests that adaptation choice across clinical acquisition settings and connects it to counterfactual composition and reversibility, showing high practical value.

AuthorsOpenReview ↗

Response to reviewer LEFC

Dear Reviewer LEFC,

We hope our rebuttal has addressed your concerns and clarified the value of our work. We would be grateful for your feedback and support for the paper’s acceptance.

Thank you!

Authors

Reviewer_M78dOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes specialisation, a way to adapt pretrained non-causal generative models into counterfactual image mechanisms for chest X-rays under limited data and distribution shift. Three variants are studied. S-SPEC adds parent conditioning to an unconditional flow model via LoRA, SA-SPEC also fine-tunes the VAE to handle acquisition shift, and M-SPEC swaps the text conditioning of a text-to-image model for structured metadata so interventions need no prompts or masks. The resulting latent flow matching mechanism, RadCF, is evaluated on CheXpert to NIH-14 and BRAX transfer against HVAE, diffusion and image-space flow baselines using axiomatic soundness metrics and KID, with claims of best realism, competitive soundness, strong data and parameter efficiency, and better results than Causal-Adapter on the same backbone. The counterfactuals are also shown to detect and reduce view shortcut learning in a downstream classifier.

Critical strengths
  1. The problem framing is sensible and practically motivated. Training deep SCMs from scratch per dataset is not realistic in medical imaging, and adapting pretrained generators under acquisition shift and measurement bias is the right question to ask.
  2. The ablations actually isolate what the paper claims. Table 2 compares S-SPEC, SA-SPEC and full fine-tuning in-domain versus under shift, and Table 8 decomposes the round-trip error into VAE reconstruction versus latent drift with paired tests. Together these make a convincing case that the SA-SPEC gain comes from fixing the autoencoder under shift rather than from extra capacity.
  3. The Causal-Adapter comparison is done on a matched backbone, with the unmatched SD1.5 numbers reported separately and the confound stated explicitly.
  4. The evaluation is unusually careful about its own weaknesses. Effectiveness is re-scored with six independent evaluators with rank agreement reported, uncertainty is propagated properly, and the appendices stress-test the Markovian assumption with linear probes on abducted noise and with deliberately confounded Morpho-MNIST and NIH setups. The authors are upfront that the latent mechanism breaks the identifiability guarantees of image-space flows.
  5. The downstream shortcut experiment is well designed. Different generators are used for augmentation and auditing to avoid circularity, and a non-generative random control is included.
Critical weaknesses
  1. Table 1 is missing the natural baseline. The from-scratch RadCF row appears only in Figure 2 and Table 9, and in Table 9 it beats SA-SPEC on effusion and sex at 100 percent data. Readers of the main table cannot see that specialisation costs effectiveness relative to training from scratch when data are plentiful.
  2. Reversibility under do(v) is the worst of all methods at 0.116, and the paper only mentions this in passing. The LoRA rank ablation in Figure 6 shows composition LPIPS around 0.09 to 0.10, far above the 0.017 reported in Table 1, with no explanation of why those numbers differ so much from the main results.
  3. The data efficiency curve is noisy. Effusion effectiveness for SA-SPEC goes 0.657 at 2 percent, 0.574 at 5 percent, 0.429 at 10 percent, 0.405 at 25 percent and back to 0.561 at 100 percent. This is non-monotonic to a degree that suggests high run-to-run variance, and no seeds or repeated runs are reported.
  4. The shortcut learning experiment has an internal inconsistency. The change rate at bias 0.95 is 0.47 in Figure 7 but 0.593 in Table 12 for what appears to be the same baseline setup.
  5. Age is treated as a continuous parent but evaluated only by MAE, and age interventions are the weakest across the board, with the do(a) MAE of 0.094 meaning errors of roughly nine years on a 100 year scale.
Questions for authors
  1. Please add the from-scratch RadCF row to Table 1 at 100 percent data. Table 9 suggests it beats SA-SPEC on effusion and sex, which changes the reading of the main comparison.
  2. Why is effusion effectiveness for SA-SPEC so low and so non-monotonic across data fractions in Table 9? Were multiple seeds run? If the variance is this large, the Figure 2 curves need error bands from repeated runs rather than propagated CIs.
  3. Composition LPIPS is about 0.09 in Figure 6 but 0.017 in Table 1 for what looks like the same configuration. What differs?
  4. The change rate at bias 0.95 is 0.47 in Figure 7 and 0.593 in Table 12. Are these the same classifier and test set?
5 comments
AuthorsOpenReview ↗

Response to Reviewer M78d (Part I)

We thank the reviewer for their positive assessment and for recognising the practical motivation and careful experimental design of our work. Below, we clarify the baselines and reported results, provide additional multi-seed experiments, and extend the evaluation of age interventions.

W1, Q1: Table 1 is missing the natural baseline.

W1: “Table 1 is missing the natural baseline. The from-scratch RadCF row appears only in Figure 2 and Table 9, and in Table 9 it beats SA-SPEC on effusion and sex at 100 percent data. Readers of the main Table cannot see that specialisation costs effectiveness relative to training from scratch when data are plentiful.”

Q1: “Please add the from-scratch RadCF row to Table 1 at 100 percent data. Table 9 suggests it beats SA-SPEC on effusion and sex, which changes the reading of the main comparison.”

R1. We apologize for the confusion and have revised the manuscript to clarify the following:

  • Table 1's “RadCF” row is the from-scratch model: RadCF trained on all NIH-14 training data without source pretraining, evaluated on the full 12,104-image test set to isolate the impact of backbones on soundness metrics. The revision changes the row “RadCF” -> “RadCF (from scratch)”.
  • Additionally, we have included RadCF SA-SPEC in Table 1 to allow a direct comparison to RadCF (from scratch).

We agree that, when all target-domain training data are available, specialisation need not uniformly improve performance: the from-scratch model achieves better effectiveness on effusion, sex and age interventions, whereas SA-SPEC improves composition and reversibility across interventions and can be more data efficient (Section 4.1, Figure 2). The revised Table 1 makes this trade-off visible (Section 4.1).

W2, Q3: Reversibility under do(v) is the worst and number discrepancy in Figure 6

W2: “Reversibility under do(v) is the worst of all methods at 0.116, and the paper only mentions this in passing. The LoRA rank ablation in Figure 6 shows composition LPIPS around 0.09 to 0.10, far above the 0.017 reported in Table 1, with no explanation of why those numbers differ so much from the main results.”

Q3: “Composition LPIPS is about 0.09 in Figure 6 but 0.017 in Table 1 for what looks like the same configuration. What differs?”

R2. We agree that reversibility under view intervention is a weakness of the from-scratch model. In the same evaluation, RadCF achieves the highest view-intervention accuracy (99.6%) and the lowest KID (0.53, scaled by ×10²). These results show strong effectiveness and realism alongside weaker cycle consistency. Additionally, view change requires a particularly sophisticated edit: switching between AP and PA changes the magnification of the heart and mediastinum, the position of the scapulae over the lung fields and the projected angle of the clavicles and ribs. Consistent with prior counterfactual literature (Ribeiro et al., 2023; Rasal et al., 2025), more effective interventions that induce larger structural changes are inherently harder to invert cleanly. We will make this trade-off explicit in the results section and identify reversibility as a limitation in Section 4.1 and Appendix B.2.

On the number discrepancy, the composition values in Figure 6 and Table 1 come from different training configurations. The composition LPIPS of 0.017 in Table 1 corresponds to RadCF trained from scratch to act as a baseline for specialisation. In contrast, Figure 6 reports the LoRA rank ablation for S-SPEC, which adapts the pretrained flow on 100% of NIH-14 while keeping the source-domain VAE frozen. The approximately 0.09–0.10 values in Figure 6 are consistent with the 0.092 composition LPIPS reported for cross-domain S-SPEC in Table 2. We hope this addresses the concern and have updated Table 1 for clarity.

AuthorsOpenReview ↗

Response to Reviewer M78d (Part II)

W3, Q2: Data efficiency curve is noisy, repeated runs

W3: “The data efficiency curve is noisy. Effusion effectiveness for SA-SPEC goes 0.657 at 2 percent, 0.574 at 5 percent, 0.429 at 10 percent, 0.405 at 25 percent and back to 0.561 at 100 percent. This is non-monotonic to a degree that suggests high run-to-run variance, and no seeds or repeated runs are reported.”

Q2: “Why is effusion effectiveness for SA-SPEC so low and so non-monotonic across data fractions in Table 9? Were multiple seeds run? If the variance is this large, the Figure 2 curves need error bands from repeated runs rather than propagated CIs.”

R3. We’ve added multi-seed experiment upon request to quantify the run-to-run variance and noise in the data efficiency curve, further supporting the specialisation’s advantage in the lower data ratio regime. Due to rebuttal compute constraints, the 100%-data from-scratch baseline is currently reported for two seeds and will be completed as soon as possible for the final version. We have left – as placeholders in the revised PDF in the meantime.

The non-monotonic effectiveness for effusion from 5% to 10% associates with substantially lower KID alongside lower effusion editing effectiveness. This non-monotonicity was also reproduced. This pattern is consistent with an attribute-dependent trade-off between editing effectiveness and realism (Ribeiro et al., 2023; Rasal et al., 2025).

Figure 2 now reports the mean and standard deviation across the four runs, and Appendix F gives the per-seed results.

W4, Q4: Number discrepancy in the shortcut learning experiment

W4: “The shortcut learning experiment has an internal inconsistency. The change rate at bias 0.95 is 0.47 in Figure 7 but 0.593 in Table 12 for what appears to be the same baseline setup.”

Q4: “The change rate at bias 0.95 is 0.47 in Figure 7 and 0.593 in Table 12. Are these the same classifier and test set?”

R4. We apologise for the confusion. The results in Figure 7 and Table 12 (now Table 20) originally used different classifier instances, trained separately for the shortcut-audit and mitigation experiments using different random seeds and tested on different methods. To make the results more intuitively comparable, we have rerun the shortcut mitigation experiments using the same checkpoints and seeds as the shortcut detection experiments.

With this revision, we also extended the utility experiment to all counterfactual generation algorithms. We compare counterfactual augmentation from S-SPEC, HVAE, diffusion and image-space flow, alongside random minority upsampling and conventional augmentation.

As shown below, our generated counterfactuals achieved overall the most stable utility when mitigating shortcut learning under highest dataset bias.

We have replaced the previous mitigation results with this updated comparison and clarified the classifier runs, test sets, and audit protocol in Appendix I. The revisions are marked in magenta.

MethodBias 0.50Bias 0.75Bias 0.95
Baseline11.0 ± 1.618.8 ± 5.546.6 ± 10.5
Random upsample24.4 ± 3.326.7 ± 3.831.1 ± 3.1
Random augmentation13.6 ± 3.017.5 ± 6.146.8 ± 10.6
CF augmentation (s-spec)10.7 ± 5.113.4 ± 3.112.7 ± 3.2
CF augmentation (HVAE)11.6 ± 2.316.1 ± 7.048.1 ± 6.7
CF augmentation (diffusion)11.5 ± 5.713.2 ± 5.425.7 ± 8.0
CF augmentation (image-space flow)11.0 ± 5.513.6 ± 5.213.5 ± 4.3

Table R2. Mean and std change rate (%) under do(v), scored on sa-spec counterfactuals (lower is better).

AuthorsOpenReview ↗

Response to Reviewer M78d (Part III)

W5: Age is treated as a continuous parent but evaluated only by MAE

W5: “Age is treated as a continuous parent but evaluated only by MAE, and age interventions are the weakest across the board, with the do(a) MAE of 0.094 meaning errors of roughly nine years on a 100 year scale.”

R5. We agree that evaluating age with only MAE did not provide a full picture and have therefore extended the evaluation to include median absolute error and directional success in Appendix B.3. Median absolute error describes the typical distance from the requested target age, while directional success measures the proportion of interventions for which the estimated age changes in the requested direction (older or younger). For reference, the pseudo-oracle’s MAE on the factual images (5.1 years in Table 4) is the error expected from the evaluator alone.

MethodMAE (years) ↓Median absolute error (years) ↓Directional success ↑
HVAE22.522.484.6%
Diffusion15.514.990.8%
Image-space flow15.214.089.7%
RadCF9.47.895.6%

Table R3. Expanded Age Intervention analysis

As indicated by the results above, precise age control remains challenging across methods. However, apart from RadCF achieving the lowest age-prediction errors among the compared methods, the additional metrics show that RadCF’s improvement extends beyond the mean error to both typical target-age accuracy and consistency of the editing direction. Due to the page limit, we have added these results in Appendix B.3 after the evaluator sensitivity to complement age evaluation.

Reviewer_HYJJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to adapt pretrained generative models for counterfactual chest X-ray generation under limited labeled data and cross-dataset distribution shifts. The authors propose a specialisation framework that enables editing of attributes such as view position, pleural effusion status, gender, and age through structured attribute conditioning, low-rank parameter updates, and VAE adaptation, and further extend the framework to text-conditioned generative models. The method is evaluated on three chest X-ray datasets in terms of generation quality, attribute control, and data efficiency, with generally strong results.

Critical strengths
  1. The paper addresses counterfactual generation for chest X-rays under limited labeled data and cross-dataset distribution shifts, which is a practically relevant problem. Existing methods often rely on relatively large amounts of target-domain data for adaptation, which can be difficult to obtain in medical imaging due to annotation costs and restrictions on data sharing. Reducing the amount of target-domain data required for adaptation is therefore an important aspect of this problem.

  2. The experimental evaluation is relatively comprehensive. It covers multiple chest X-ray datasets, different types of interventions, and varying amounts of training data. The authors also provide ablation studies and downstream-task evaluations, offering evidence for the effectiveness and practical utility of the proposed approach from several perspectives.

Critical weaknesses
  1. The methodological novelty should be clarified further. S-SPEC introduces parameter-efficient adaptation through LoRA [1]; the joint optimization of the VAE and generative network in SA-SPEC builds on the training strategy of REPA-E [2]; and M-SPEC incorporates structured attribute conditioning into a pretrained generative model, which is also conceptually close to the adaptation strategy used in Causal-Adapter [3]. While combining these techniques to address data scarcity and distribution shift in medical imaging is meaningful, the key methodological advances over these prior approaches are not yet sufficiently clear. The authors should better distinguish the proposed components from existing methods and clarify which technical designs constitute the unique contributions of this work.

  2. Table 1 shows that RadCF performs worse than some comparison methods in preserving non-target attributes under several interventions. For example, under the gender intervention, the pleural effusion preservation accuracy is 0.807, compared with 0.887 for image-space flow. Under the age intervention, preservation performance for both pleural effusion and gender also decreases. These results may indicate that stronger target editing is accompanied by unintended changes in non-target attributes. Although the paper discusses the general trade-off between intervention effectiveness and identity preservation, these specific differences are not analyzed in sufficient detail. Further discussion is needed regarding the source of these changes and the relationship between stronger target editing and preservation of non-target attributes.

[1] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).

[2] Leng, Xingjian, et al. "Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[3] Tong, Lei, et al. "Causal-adapter: Taming text-to-image diffusion for faithful counterfactual generation." arXiv preprint arXiv:2509.24798 (2025).

Questions for authors
  1. Compared with existing approaches such as LoRA [1], REPA-E [2], and Causal-Adapter [3], what are the key technical advances of S-SPEC, SA-SPEC, and M-SPEC, respectively? Beyond organizing these components into a unified framework for medical image adaptation, which specific designs should be regarded as methodological contributions unique to this work?
  2. How do the authors explain the decrease in several non-target attribute preservation metrics in Table 1? For example, under the gender intervention, RadCF achieves a pleural effusion preservation accuracy of 0.807, compared with 0.887 for image-space flow. Do these differences indicate that stronger target editing leads to more unintended changes in non-target attributes, or might they instead arise from factors such as the evaluation classifier? Please further clarify the trade-off between target editing effectiveness and non-target attribute preservation.
  3. How do the authors explain the non-monotonic behavior of SA-SPEC in Table 9? For example, the pleural effusion editing success rate decreases from 0.574 with 5% of the training data to 0.429 with 10%. How were the data subsets and training budgets determined for different data fractions? Are results across different random seeds or independently sampled subsets available to support the stability of the main data-efficiency conclusion?
3 comments
AuthorsOpenReview ↗

Response to Reviewer HYJJ (Part I)

We thank the reviewer for recognising the practical relevance of our work and the breadth of our evaluation across datasets, interventions and data regimes. Below, we clarify the methodological contribution, analyse non-target preservation, and report multi-seed results for the data-efficiency experiments.

W1, Q1: The methodological novelty should be clarified further

W1: “The methodological novelty should be clarified further. S-SPEC introduces parameter-efficient adaptation through LoRA; the joint optimization of the VAE and generative network in SA-SPEC builds on the training strategy of REPA-E; and M-SPEC incorporates structured attribute conditioning into a pretrained generative model, which is also conceptually close to the adaptation strategy used in Causal-Adapter. … The authors should better distinguish the proposed components from existing methods and clarify which technical designs constitute the unique contributions of this work.”

Q1: “Compared with existing approaches such as LoRA, REPA-E, and Causal-Adapter, what are the key technical advances of S-SPEC, SA-SPEC, and M-SPEC, respectively? Beyond organizing these components into a unified framework for medical image adaptation, which specific designs should be regarded as methodological contributions unique to this work?”

R1. Our primary contribution is a principled framework for adapting pretrained image generators, conditional or unconditional, to counterfactual generation under a range of clinical domain mismatches. This operationalised the perspectives by Castro et al. (2020) on the causal analysis of dataset shift in medical imaging, and by Finlayson et al. (2021) on the need for mitigation strategies for different clinical shifts.

Earlier counterfactual methods leave this dataset shift problem largely open (Ribeiro et al., 2023; Rasal et al., 2025; Ribeiro et al., 2025). They learn mechanisms for specified datasets and causal graphs. Existing medical adaptation and editing methods address other parts of this problem. Yeganeh et al., 2025 and Tong et al., 2026 treat adaptation of a general image generator to a target clinical dataset as an overall task, without systematically distinguishing how differences in causal parents, image acquisition and conditioning representation affect adaptation. RadEdit addresses out-of-domain editing through inference-time spatial constraints (Pérez-García et al., 2025a). Notably, all three retain a fixed pretrained VAE in their reported implementations, and so do not examine when the image representation itself needs adaptation across clinical domains.

Thus, to the best of our knowledge, we are the first to formulate and experimentally evaluate a framework that jointly connects structural, acquisition and measurement shifts to the adaptation requirements of pretrained generators for SCM-guided counterfactual generation across clinical datasets.

Our framework relates domain shifts to concrete optimisation strategies for model adaptation. For example, it explains why increasing flow transport capacity alone fails under acquisition shift without latent space adaptation (Section 4.1 When is SA-SPEC necessary, Table 2) and is generalisable to different backbones (Section 4.2 Causal Text-to-Image Generation). The interactions between shifts and components also allow our method to outperform recent works (Ribeiro et al. 2025; Tong et al., 2026) in effectiveness and realism (see in Table1, Table 3 and Appendix H), showing that our carefully constructed approach provides advantages. Overall, these experiments establish the practical value and scope of specialisation.

AuthorsOpenReview ↗

Response to Reviewer HYJJ (Part II)

W2, Q2: Further discussion is needed regarding … stronger target editing and preservation of non-target attributes.

W2: “Table 1 shows that RadCF performs worse than some comparison methods in preserving non-target attributes under several interventions. For example, under the gender intervention, the pleural effusion preservation accuracy is 0.807, compared with 0.887 for image-space flow. Under the age intervention, preservation performance for both pleural effusion and gender also decreases. … Further discussion is needed regarding the source of these changes and the relationship between stronger target editing and preservation of non-target attributes.”

Q2: “Do these differences indicate that stronger target editing leads to more unintended changes in non-target attributes, or might they instead arise from factors such as the evaluation classifier? Please further clarify the trade-off between target editing effectiveness and non-target attribute preservation.”

R2. While we acknowledge this difference in non-target preservation, it is consistent with the trade-off between intervention effectiveness and identity preservation discussed in prior work (Monteiro et al., 2023; Rasal et al., 2025). This also mirrors attribute amplification (Xia et al., 2024), where high effectiveness exaggerates attributes that should remain unchanged. Analysing this is fertile ground for future work.

To further show that this phenomenon does not arise from evaluation classifier, we re-evaluated do(sex) and do(age) on the NIH-14 test set using another effusion classifier (Pérez-García et al., 2025b; AUROC = 0.85, Appendix B).

To prevent failed target edits from artificially inflating preservation, do(sex) is evaluated on the n = 6,615 subset where all four methods successfully flipped sex, determined by our pseudo-oracle classifier; do(age) uses the full test set (n = 12,104) since binary success thresholding is not applicable to continuous variables.

InterventionImage-FlowRadCF from scratchDiffusionSA-SPEC
Sex3.6% [3.2,4.1]6.7% [6.1,7.3]5.5% [4.9,6.0]5.6% [5.1,6.2]
Age3.0% [2.7,3.4]5.9% [5.5,6.4]3.9% [3.4,4.4]4.0% [3.7,4.4]

Table R1. Effusion prediction-change rate; lower is better. Intervals show 95% CI using bootstrapping.

The independent evaluator shows a similar pattern for both interventions and matches the observations in Table 1. Thus, our discussion in Section 4.1 about the trade-off between intervention effectiveness and non-target preservation still holds. We have added independent-evaluator results to Appendix B.2 and added discussions with attribute amplification.

Q3: non-monotonic behavior of SA-SPEC in and multi-seed experiments

Q3: “How do the authors explain the non-monotonic behavior of SA-SPEC in Table 9? For example, the pleural effusion editing success rate decreases from 0.574 with 5% of the training data to 0.429 with 10%. How were the data subsets and training budgets determined for different data fractions? Are results across different random seeds or independently sampled subsets available to support the stability of the main data-efficiency conclusion?”

R3. The non-monotonic effectiveness for effusion from 5% to 10% associates with substantially lower KID alongside lower effusion editing effectiveness. This pattern is consistent with an attribute-dependent trade-off between editing effectiveness and realism (Ribeiro et al., 2023; Rasal et al., 2025). The data subsets are randomly selected for different data fraction, and the training budgets are scaled with dataset size based on the observed stabilisation of the training loss.

Upon request, we repeated the data-efficiency experiments with three additional random seeds, to ensure our data-efficiency conclusions are robust to training seeds. Due to rebuttal compute constraints, the 100%-data from-scratch baseline is currently reported for two seeds and will be completed as soon as possible for the final version. We have left – as placeholders in the revised PDF in the meantime.

Across all multi-seed runs, our core conclusions hold with low variance: SA-SPEC outperforms training from scratch in the low-data regime (particularly at 5% data), while maintaining superior counterfactual composition and reversibility at larger data fractions.

Figure 2 now reports the mean and standard deviation across the four runs, and Appendix F gives the per-seed results.

AuthorsOpenReview ↗

Response to reviewer HYJJ

Dear Reviewer HYJJ,

We hope our rebuttal has addressed your concerns and clarified the value of our work. We would be grateful for your feedback and support for the paper’s acceptance.

Thank you!

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to AC and all reviewers

Dear Area Chair and reviewers,

We thank the reviewers for their thoughtful reviews. We are encouraged that all three reviewers recommended weak acceptance and recognised the practical importance of adapting counterfactual image generators to medical imaging settings with limited data and distribution shift. We also appreciate the reviewers’ positive assessment of our experimental design, which they found comprehensive (HYJJ, LEFC) and well designed to support our claims (M78d).

We have carefully addressed all the comments that were raised. The main changes are summarised here, answered in more depth in the individual responses, and the PDF has been revised accordingly (marked in magenta for your convenience).

  1. Methodological contribution. We clarify how our framework connects structural, acquisition and measurement shifts to adaptation choices, leading to practical and clinical benefits which previous works lack (HYJJ, LEFC).
  2. Data efficiency and repeated runs. We added experiments with additional random seeds, per-seed results and standard-deviation bands in Figure 2. Due to rebuttal compute constraints, the 100%-data from-scratch baseline is temporarily reported for two seeds and will be updated as soon as possible for the final version (HYJJ, M78d).
  3. Target editing and preservation. We provide additional analyses of the relationship between target editing and non-target attribute preservation and further discuss the trade-off between editing strength and identifiability (HYJJ).
  4. Age Evaluation. We have extended the age evaluation to include median absolute error and directional success alongside MAE (M78d).
  5. Preserving unmodelled pathological findings. We measure how often predictions for unmodelled diseases change under view interventions as an indication of clinical faithfulness (LEFC).

We are happy to answer further questions during the discussion period.

Kind regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.