acceptodds
Under review as a conference paper at ICLR 2027

IWCAE: Importance-Weighted Conditional Autoencoders for Domain-Adaptive Generative Modeling

Abstract

Generative models characterize full data distributions and support a broad range of downstream tasks, but their performance may deteriorate when the training and testing distributions differ. Existing domain adaptation methods are often developed under covariate shift or label shift assumptions, which may be restrictive under substantial domain shift. We instead consider a more flexible setting in which there exists a representation under which the conditional response distribution is invariant across domains. We propose an importance-weighted conditional variational autoencoder, termed IWCAE, for transferable conditional distribution learning. IWCAE learns the full conditional response distribution through an importance-weighted conditional likelihood objective while using a maximum mean discrepancy penalty to align the marginal representation distributions across the training and testing domains. This design seeks to preserve predictive information while suppressing domain-specific variation. We further develop a sampling-importance-resampling procedure for approximate predictive sampling when direct sampling from the conditional latent prior is difficult. Theoretically, we establish the asymptotic validity of the proposed sampling procedure. Experiments on three medical datasets demonstrate the effectiveness of IWCAE in improving cross-domain predictive performance and quantifying predictive uncertainty under distribution shift.

Primary area:
generative models
Keywords:
domain adaptation, conditional generative modeling, variational autoencoders, invariant representation learning, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_gJ4EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers the problem of learning conditional generative models under changes in the train and test distributions. They propose a methodology called importance-weighted conditional variational autoencoder (IWCAE), together with learning and inference algorithms. For sampling–importance–resampling inference, they prove that as the number of samples tends to infinity, their algorithm's samples converge in distribution to the desired distribution. Experiments using their method are then performed in three different domains with different data modalities, showing superior performance compared with some baselines.

Critical strengths
  • The problem is clearly interesting and important.
  • The central proposal and its motivating modeling assumption (Conditional invariant representation) appears sound and reasonable.
  • The results appear to suggest some efficacy of the proposed method.
Critical weaknesses
  • Relation to literature: the paper assumes a lot of background in generative modeling, and in particular how the new method compares with existing works is not clearly described for those unfamiliar.
  • Discussion of experiments in introduction: there is no discussion of the experimental results in the introduction other than a single sentence in the 'contributions' list. There should be such discussion. It also makes it harder to assess what the central claims or conclusions of the experiments are.
  • Exposition of what the experiments show is lacking even in the later sections. For example, what are the methods against which you compare? You simply list them without discussing how they relate to your method.
Questions for authors

List of comments:

  • “Experiments on three medical datasets with heterogeneous data modalities the effectiveness of IWCAE for conditional generative prediction under cross-domain distribution shifts.” Has some grammatical typo.
  • The introduction does not discuss your experiments at all until there is one sentence about them in the ‘contributions’ list. They should be discussed in the introduction
  • “where testing responses are unavailable during training” This is slightly confusing, since, presumably the testing inputs are not available during training.
  • The phrase “is related to invariant representation assumptions commonly used in domain adaptation” seems to be claiming an important relationship of your work to existing work, but is not discussed in any more detail. Can this relationship be explained in more detail?
  • “We therefore introduce a latent variable Z to capture the residual information in X that is not encoded by τ (X)” This doesn’t make sense to me. What is the difference between \tau(X) and Z. How do their responsibilities differ. In fact, in equation (2), it seems to me that the only way for information in XX to flow to ZZ is through the representation τ(X)\tau(X). In other words, there is no way for Z to contain information about XX other than that contained in τ(X)\tau(X).
3 comments

We sincerely thank the reviewer for the constructive comments and valuable suggestions. We provide point-by-point responses to the raised concerns and questions below.

Response to Critical Weaknesses

1: We agree that a clearer discussion of the relationship between IWCAE and existing methods would help readers better understand the methodological contributions of our work.

Our framework is closely related to three lines of research: conditional variational autoencoders (CVAEs), importance-weighted variational inference, and invariant representation learning for domain adaptation. Our primary goal is to extend conditional generative modeling to settings involving domain shift by learning an invariant representation of the predictors.

A key methodological distinction of IWCAE lies in incorporating an importance-weighted variational approximation into conditional distribution learning under domain shift. Since CVAEs aim to model conditional outcome distributions, incorporating an importance-weighted training objective provides a tighter variational lower bound on the conditional log-likelihood, potentially improving conditional distribution estimation given the learned invariant representation. By jointly optimizing this objective and representation alignment, IWCAE integrates importance-weighted conditional generative modeling with invariant representation learning to facilitate distributional prediction under domain shift.

2: We have added a discussion of the experiments to the Introduction, immediately following the description of the proposed IWCAE framework, highlighting the applicability of IWCAE across different medical data modalities.

3: Due to the page limit, we initially provided only brief descriptions of the compared methods in the main text, with additional details on the construction of IWCAE and the architectures of the baseline methods presented in Appendix A.3.

For the first two experiments, the first compared method serves as the base prediction model, while the remaining four methods are domain adaptation approaches built upon the same base architecture. These comparisons allow us to evaluate the effectiveness of IWCAE relative to both the original prediction model and existing domain adaptation methods. For the third experiment, we select four established models for 4D Flow MRI super-resolution to evaluate the predictive performance of IWCAE under domain shift. Among these methods, 3D U-Net serves as the base architecture for both DSR and IWCAE, while 4DFlowNet is another established architecture that has demonstrated strong performance in 4D Flow MRI super-resolution, with Bagging representing an ensemble-based variant of 4DFlowNet. Since training on computational fluid dynamics (CFD) data and testing on 4D Flow MRI data is a commonly adopted setting for this task, all four baseline methods are evaluated under the same simulation-to-real domain shift.

We have expanded Appendix A.3 to provide more detailed descriptions of the baseline architectures and explain how they relate to IWCAE.

Response to Questions

1: Thank you for pointing out this grammatical error. We have corrected the sentence in the revised manuscript.

2: We have added a brief discussion of the experiments in the Introduction, immediately following the description of the proposed IWCAE framework, to better highlight its applicability across different medical data modalities.

3: Our work considers an unsupervised domain adaptation setting, where labeled samples from the training domain and unlabeled inputs from the testing domain are available during training, while the corresponding testing-domain responses remain unobserved. The unlabeled testing-domain inputs are used to facilitate distributional alignment between the training and testing domains.

This setting is particularly relevant to medical data analysis, where unlabeled data are often abundant, but obtaining corresponding ground-truth labels or responses can be costly and time-consuming. For example, low-resolution medical images may be readily available, whereas acquiring corresponding high-resolution reference images requires additional imaging resources. Similarly, large collections of clinical notes may be available, but obtaining reliable diagnostic labels requires expert annotation. These practical challenges motivate our consideration of unsupervised domain adaptation.

Response to Questions

4: Our assumption is closely related to invariant representation learning in domain adaptation, where a transformation τ(X)\tau(X) is sought to preserves prediction information while reducing the discrepancy between representations across domains. The mapping τ\tau may represent a low-dimensional linear or nonlinear transformation of the input or, when τ\tau is a masking operator, a subset of the original covariates (Pan et al., 2010; Long et al., 2014; Peng et al., 2019; Wu et al., 2025). Since our assumption accommodates these commonly used forms of invariant representations, it provides a more general formulation encompassing different representation structures. We have revised the statement to Section 3.1 to clarify the relationship between our assumption and existing domain adaptation approaches.

5: In our framework, τ(X)\tau(X) represents the observed predictors, whereas ZZ is a latent variable introduced to capture residual variation in YY that is not fully explained by τ(X)\tau(X). Such information may include the effects of predictive factors in XX that are not preserved by the transformation τ\tau.

Specifically, as shown in Eq (2), the posterior pθ(z∣τ(x),y)p_{\theta}(\mathbf z \mid \tau(\mathbf x),\mathbf y) is conditioned on both the transformed predictors and the observed response. Since the response may depend on components of XX that are not retained by τ(X)\tau(X), conditioning on y\mathbf y can provide additional information about these omitted components, allowing ZZ to represent their contributions to the conditional outcome distribution.

For example, consider X=(X1,X2)X=(X_1,X_2) with τ(X)=X1\tau(X)=X_1 and Y=X1+X2+εY=X_1+X_2+\varepsilon. Although X2X_2 is not included in τ(X)\tau(X), it contributes to YY. Consequently, the posterior distribution of ZZ given (X1,Y)(X_1,Y) can capture information associated with X2X_2 through its contribution to YY, allowing the model to characterize residual variability that is not fully explained by τ(X)\tau(X).

We have revised the corresponding statement to clarify this distinction.

Dear Reviewer,

We understand that you have a busy schedule. As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal and share any further thoughts or concerns. Thank you again for your constructive feedback, which has helped us substantially improve our manuscript.

Reviewer_uH4GOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper addresses the problem of distribution shift at test time by assuming there exists a transformation of the predictor such that the marginal and conditional distributions using the transformer predictors are equal (no shift). They use this assumption to propose a model based on a conditional VAE + an importance sampling technique to better estimate the marginal likelihood. They train the model using a standard NLL objective but penalize for distance between train and test marginal distributions. They demonstrate their approach on 3 medical applications (dementia classification, MRI reconstruction, and super-resolution of 4dflow MRI data), and their method improves over each baseline on average (baselines are both general methods and methods specifically aimed at domain shift).

Critical strengths

Interesting combination of existing methods to address domain shift in generative modeling and classification in medical domains, applied in 3 different medical domains.

Critical weaknesses
  • The performance improvements over baselines in all three experimental settings are quite small (especially relative to the variation across test images for each method) and no statistical significance is reported, making it difficult/impossible to interpret the practical significance of this method.
  • No ablations of the core method are applied, making it impossible to judge whether each is necessary
  • The paper seems to miss some more recent work and mostly discusses relatively old works.
  • How the proposed baselines compare in terms of parameter count, training compute, and inference cost is not exactly discussed.
Questions for authors
  • why do you need algorithm 1 to sample from the predictive distribution (seems like the conditional latent prior is Gaussian and can be sampled from directly (Appendix A.2, Equation 6) unless I'm missing something)?
  • Can the authors situate their work in more recent work? Most of the citations in the introduction and related work are relatively old. For example, how does the approach compare to recent diffusion/flow-based approaches? Would any of these be appropriate baselines for the tasks considered?
  • Table 1 is hard to interpret without error bars/confidence intervals. For example, the in-domain improvement is quite small (especially given the test set size of 571 images).
  • How many test examples do the OOD1 and OOD2 test sets contain for dementia classification in 5.1? Does the 918 number refer to the combined OOD1 and OOD2, or does each have 918 samples?
  • Can the authors ablate parts of their method, such as the alignment part of the objective, and the SIR sampling procedure?
  • Can the authors report how the proposed method compares with the baselines in terms of parameter count, training compute, and inference cost?
  • In table 2, it seems like the improvements over baselines are small relative to variation across test images. Could the authors report confidence intervals or paired statistical significance tests for these improvements, and comment on their practical significance?
  • For Section 5.3, could the authors report the numerical MSE results, including variability and confidence intervals for the improvements over the baselines?
7 comments

We sincerely thank the reviewer for the constructive comments and valuable suggestions. We provide point-by-point responses to the raised concerns and questions below.

Response to Critical Weaknesses

1: IWCAE demonstrates competitive or improved predictive performance across three distinct tasks and data modalities, suggesting its applicability to different domain-shift scenarios. In contrast, the baseline methods are primarily designed for specific tasks. To further assess the reliability of the observed improvements, we have added 95% exact binomial confidence intervals for dementia classification and paired tt-tests between IWCAE and second strongest baseline models for MRI reconstruction and 4DF super-resolution. All reported paired comparisons yield P<0.001P < 0.001. These additional analyses provide statistical evidence supporting the observed performance gains.

2: We have added comprehensive ablation studies in Section 5.4 to evaluate the contributions of importance weighting and MMD-based alignment. The results, summarized in Table 3, show that the full IWCAE framework with K=5K=5 and MMD alignment achieves the best OOD performance across all three tasks, supporting the effectiveness of these components.

3: We have expanded the Introduction and Related Work sections to incorporate more recent studies and better position IWCAE within the existing literature.

Regarding the experiments, our primary objective is to demonstrate the applicability of IWCAE across different medical data analysis tasks and domain-shift settings. We therefore selected representative and strong task-specific baseline methods, including recent approaches, for each application. The experimental results demonstrate competitive or improved performance across all three tasks, particularly under domain shift, supporting the general applicability of the proposed IWCAE framework.

4: We have added detailed comparisons of parameter counts, training time, and inference time across the three experiments in Sections A.3.1, A.3.2, and A.3.3 of the Appendix. Although IWCAE generally requires more trainable parameters and longer training time than the corresponding baselines, its inference cost remains comparable to that of vanilla CVAE. These results illustrate the trade-off between computational cost and improved predictive performance under domain shift.

Response to Questions

1: The direct sampling approach is applicable when the prior is specified as a Gaussian distribution. However, if we want to adopt more flexible prior distributions, direct sampling from the resulting conditional prior may no longer be feasible. We therefore introduce SIR as a flexible proposal-based inference strategy for settings in which direct sampling from the conditional prior is difficult, while evaluation of the corresponding density remains tractable. Since our goal is to develop a broadly applicable methodology and software package that can accommodate a wide range of practical applications, we do not want to restrict users to Gaussian priors. Implementing the more general SIR procedure therefore allows our method and software to accommodate a broader class of prior specifications and modeling needs. We have clarified this distinction in Section~3.4.

2: We have expanded the Introduction and Related Work sections to discuss more recent developments, including diffusion/flow-based generative models, and clarify their relationship to our approach. Diffusion/flow-based models are powerful alternatives to VAEs for generative modeling, each with distinct advantages and computational characteristics. However, the primary focus of our experiments is to evaluate the benefits of importance-weighted conditional distribution learning and invariant representation learning under domain shift, rather than to compare different generative model families. We believe that the underlying principles of IWCAE, including importance-weighted learning and invariant representation alignment, could also be extended to diffusion and flow-based generative models, although the specific implementation would require further investigation.

For the experiments, we select task-specific baselines under the corresponding domain-shift settings. For MRI reconstruction and 4D Flow MRI super-resolution, we include strong task-specific baselines, such as VarNet, 4DFlowNet, and the more recent ensemble-based Bagging approach, to evaluate IWCAE against established methods under domain shift. These comparisons provide empirical evidence of the applicability of IWCAE under diverse distribution shifts.

Response to Questions

3: To better quantify the uncertainty in the reported classification performance, we have added 95% exact binomial confidence intervals for all methods in Table 1 of the revised manuscript. For OOD1, IWCAE achieves an accuracy of 0.790 (95% CI: 0.768–0.810), compared with 0.752 (95% CI: 0.729–0.774) for MDD. Similarly, for OOD2, IWCAE achieves an accuracy of 0.753 (95% CI: 0.724–0.780), compared with 0.703 (95% CI: 0.672–0.732) for MDD. These results indicate that the observed accuracy improvements are pronounced under domain shift, consistent with the primary objective of IWCAE.

4: The OOD2 test set contains 918 clinical notes from non-neurology departments, whereas the OOD1 test set contains 1489 clinical notes, comprising the same 918 notes from non-neurology departments and additional 571 notes from the neurology department. Thus, OOD2 is a subset of OOD1 and represents a more intense domain shift, as it consists exclusively of clinical notes from non-neurology departments.

5: We have conducted additional ablation studies to investigate the contributions of importance weighting and representation alignment, with the results summarized in Table 3 of Section 5.4. The full IWCAE framework achieves the best OOD predictive performance across all three tasks, providing empirical evidence of the benefits of these components.

We have also compared SIR with direct prior sampling. Since our current implementation uses a Gaussian conditional prior, the two approaches exhibit comparable predictive performance. As clarified in our response to the related question, SIR is introduced as a flexible proposal-based inference strategy that can accommodate more complex conditional prior distributions where direct sampling may be difficult.

6: In our experiments, we use the same backbone architectures for IWCAE and the corresponding baseline models to ensure comparable experimental settings. Compared with non-CVAE-based models, IWCAE incorporates additional encoders for latent-variable modeling. Compared with vanilla CVAE, IWCAE further introduces invariant representation learning and an importance-weighted training objective. During inference, SIR requires proposal sampling, importance-weight calculations, and resampling, although these operations incur relatively modest overhead due to the low dimensionality of the latent variable ZZ.

Taking dementia classification as an example, we summarize the model complexity and computational costs in the following table. Specifically, CVAE and IWCAE have 516,131 and 616,803 trainable parameters, respectively. Their training times over 50 epochs are 88.01 s and 155.87 s, respectively, while their total inference times across the three evaluation settings are 5.82 s and 6.99 s. Thus, although IWCAE incurs additional training costs, its inference time is only approximately 20% higher than that of CVAE. Importantly, IWCAE improves dementia classification accuracy from 0.737 to 0.790 on OOD1 and from 0.682 to 0.753 on OOD2 compared with vanilla CVAE. These results illustrate the trade-off between computational cost and improved predictive performance under domain shift. The comparisons of model complexity and computational costs for the other two experiments are presented in Sections A.3.2 and A.3.3, respectively.

MetricCNNDANNMDDCNN+EngCVAEIWCAE
Parameter count147,650151,940147,780147,650516,131616,803
Training time (s/epoch)1.152.512.051.521.763.12
Total inference time (s)0.530.490.547.395.826.99

7: Since all methods were evaluated on the same MRI images, we performed paired tt-tests to assess the statistical significance of the observed improvements. For in-domain knee MRI, IWCAE achieves higher SSIM and lower NMSE than VarNet on 100% and 96% of the evaluated images, respectively. For OOD brain MRI, IWCAE achieves higher SSIM and lower NMSE than CVAE on 81.6% and 57.0% of the evaluated images, respectively. All four comparisons yield P<0.001P < 0.001. These findings provide statistical evidence supporting the observed improvements of IWCAE.

Response to Questions

8: We have added numerical MSE results for both normal and aneurysmal vascular geometries, reported as mean and standard deviation across 30 evaluations. We further quantified the improvements over DSR using mean paired differences, 95% confidence intervals, and paired tt-tests. The results are presented in Table 2. These additional analyses provide quantitative evidence supporting the improvements achieved by IWCAE, while highlighting variations across velocity components and vascular geometries.

Table 2: MSE comparison between IWCAE and DSR for normal and aneurysmal vascular geometries across 30 evaluations. MSE values are mean (SD); the mean improvement is DSR minus IWCAE, followed by its 95% bootstrap CI. All MSE-related values are in units of 10−210^{-2}. P-values are from paired tt-tests.

GeometryMetricIWCAE MSEDSR MSEMean improvement (95% CI)PP-value
NormalX0.953 (0.058)0.855 (0.062)-0.098 (-0.111, -0.086)$ < 0.001$
NormalY1.636 (0.113)1.739 (0.139)0.103 (0.090, 0.116)$ < 0.001$
NormalZ0.873 (0.033)0.918 (0.043)0.045 (0.037, 0.053)$ < 0.001$
NormalMagnitude2.449 (0.162)2.856 (0.187)0.407 (0.389, 0.426)$ < 0.001$
AneurysmX0.245 (0.010)0.263 (0.013)0.018 (0.015, 0.021)$ < 0.001$
AneurysmY0.745 (0.027)0.761 (0.028)0.016 (0.011, 0.021)$ < 0.001$
AneurysmZ0.339 (0.014)0.395 (0.017)0.056 (0.053, 0.060)$ < 0.001$
AneurysmMagnitude0.626 (0.025)0.784 (0.031)0.157 (0.151, 0.164)$ < 0.001$
Reviewer_4Q1rOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IWCAE for learning conditional outcome distributions under domain shift. The method combines an importance-weighted conditional objective with MMD-based alignment of source and target representations. The paper also introduces SIR procedure for predictive sampling. The authors provide results for the sampling procedure and evaluate the framework on dementia classification, MRI reconstruction, and 4D Flow MRI super-resolution.

Critical strengths

The problem is practically relevant. Learning conditional outcome distributions while adapting to unlabeled target inputs is a useful direction, particularly for applications where prediction uncertainty matters. Also, the reported results look promising.

Critical weaknesses
  1. The contributions of individual components are insufficiently stated. The experiment misses direct ablation of IWCAE without MMD and IWCAE without importance weighting, and it is also unclear whether the CVAE+MMD baseline corresponds to an identical version of IWCAE with K=1. Thus, how much each component is contributing and whether those components are necessary are not fully demonstrated in the ablation study.

  2. The uncertainty estimates are not directly validated for calibration in the target domain. Figures 3 show that uncertainty is associated with prediction quality, but this itself does not establish calibration. For example, MRI experiment does not report whether the 95% prediction intervals actually cover 95% of the corresponding ground-truth pixel values.

  3. The necessity of the SIR inference procedure is unclear. Appendix A specifies a diagonal Gaussian conditional prior that can be sampled from directly. Sampling a latent variable from this prior and then an output from the decoder already gives a sample from model's predictive distribution. It is confusing whether proposal encoder and SIR procedure are necessary beyond this direct sampling approach.

Questions for authors
  1. Could you provide an IWCAE ablation without MMD and importance weighting, respectively, and clarify whether CVAE+MMD is identical to IWCAE with K=1?
  2. Could you report the empirical coverage and width of the 95% prediction intervals on target-domain data?
  3. What advantage does SIR offer over direct prior sampling?
  4. What values of L and M were used in Algorithm 1 for each experiment?
5 comments

We sincerely thank the reviewer for the constructive comments and valuable suggestions. We provide point-by-point responses to the raised concerns and questions below.

Response to Critical Weaknesses

1: We have added comprehensive ablation studies in Section 5.4 based on the configurations summarized in the following table. The results show that both components contribute to improved OOD predictive performance across all three tasks. We have also expanded the description of the CVAE+MMD baseline to clarify that it differs from IWCAE with K=1K=1.

ConfigurationKKRepresentationMMD alignmentImportance weighting
Vanilla CVAE1τ(X)=X\tau(X)=XNoNo
IWCAE (K=1K=1)1LearnedYesNo
IWCAE (No alignment)5τ(X)=X\tau(X)=XNoYes
Full IWCAE5LearnedYesYes

2: We would first like to clarify that our work does not claim guaranteed uncertainty calibration under domain shift. When there is no domain shift, the importance-weighted objective provides a tighter variational lower bound, which provides more accurate estimation of the conditional distribution. In our simulation experiments without domain shift, IWCAE achieves empirical coverage close to the nominal levels, demonstrating its ability to provide reliable uncertainty estimates in the evaluated settings. However, under domain shift, achieving nominal coverage is more challenging, as it depends on the accuracy of the learned invariant representation and the estimated target-domain conditional distribution.

To further evaluate empirical coverage on real data, we consider the dementia classification experiment with binary outcomes. At the nominal 95% level, IWCAE achieves empirical coverage rates of 0.963 for in-domain testing and 0.919 and 0.894 for OOD1 and OOD2, respectively. These results demonstrate near-nominal coverage in the in-domain setting and illustrate the challenges of maintaining nominal coverage under distribution shift. Note that we do not use the MRI example to evaluate coverage probability because image pixels exhibit spatial dependence. Consequently, simply averaging the coverage indicators across pixels does not provide an appropriate measure of coverage probability.

3: The direct sampling approach is applicable when the prior is specified as a Gaussian distribution. However, if users wish to adopt more flexible prior distributions, direct sampling from the resulting conditional prior may no longer be feasible. We therefore introduce SIR as a flexible proposal-based inference strategy for settings in which direct sampling from the conditional prior is difficult, while evaluation of the corresponding density remains tractable. Since our goal is to develop a broadly applicable methodology and software package that can accommodate a wide range of practical applications, we do not want to restrict users to Gaussian priors. Implementing the more general SIR procedure therefore allows our method and software to accommodate a broader class of prior specifications and modeling needs. Furthermore, as we aim to develop IWCAE into a modular framework applicable to different backbone models, adopting SIR as an optional inference procedure offers additional flexibility. We have clarified this distinction in Section 3.4.

Response to Questions

1: We have added Table 3 to Section 5.4 of the main text to present the comprehensive ablation results. The results show that the full IWCAE framework achieves the best performance across all OOD settings in the three tasks.

CVAE+MMD is not equivalent to IWCAE with K=1K=1. Specifically, CVAE+MMD combines a vanilla CVAE, without importance-weighted learning or invariant representation learning, with MMD-based alignment between the estimated predictive distributions P^θ^tr(⋅∣X)\hat{P}^{\mathrm{tr}}_{\widehat{\theta}}(\cdot \mid X) and P^θ^te(⋅∣X)\hat{P}^{\mathrm{te}}_{\widehat{\theta}}(\cdot \mid X). The comparison shows that IWCAE achieves better performance than CVAE+MMD.

2: We thank the reviewer for this helpful suggestion. As discussed in our response to the related concern, IWCAE achieves empirical coverage close to the nominal level in the in-domain testing, whereas maintaining nominal coverage under domain shift is more challenging.

For the dementia classification experiment, since the response is binary, we evaluate prediction sets rather than conventional prediction intervals. At the nominal coverage level of 95%, the empirical coverage rates for OOD1 and OOD2 are 0.919 and 0.894, respectively, with corresponding average prediction-set sizes of 1.422 and 1.483. At the nominal coverage level of 90%, the empirical coverage rates are 0.893 and 0.870, with corresponding average prediction-set sizes of 1.323 and 1.374.

Response to Questions

3: We thank the reviewer for this important question. Our motivation for introducing SIR follows established proposal-based inference approaches in deep latent-variable models (Sohn et al., 2015; Mattei and Frellsen, 2019). In particular, Sohn et al. (2015) demonstrated that importance sampling can improve the efficiency of conditional likelihood estimation, while Mattei and Frellsen (2019) employed SIR to generate samples from conditional distributions in latent-variable models. An important advantage of proposal-based inference is its flexibility in accommodating complex conditional prior distributions for which direct sampling may be difficult but density evaluation remains feasible. Although direct prior sampling is feasible for the Gaussian prior considered in our experiments, SIR provides a more general inference strategy. We have clarified this motivation and the role of SIR in Section 3.4.

4: For the first two experiments, we set L=128L=128 and M=64M=64, while for the third experiment, we set L=100L=100 and M=50M=50. We have added these settings to Appendix A.3 of the revised manuscript.

Dear reviewer

We understand you have a busy schedule, but with the deadline approaching, we would like to begin the discussion with you. We truly appreciate all your feedback, which has greatly strengthened the paper.

Reviewer_Yn2pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed Importance-Weighted Conditional Autoencoder (IWCAE) to model the label distributions for data from unseen target domains. IWCAE combines conditional importance-weighted variational objective and MMD penalty to aligns source and target representation distributions. At inference time, the authors proposed sampling–importance–resampling (SIR) and shows that SIR prediction is asymptoticly correct for the fitted model. Empirically, the authors evaluate their method on dementia classification, MRI reconstruction, and 4D Flow MRI super-resolution.

Critical strengths
  1. The paper addresses an important and relevant problem, as distribution shifts arise in many real-world applications. Learning the full conditional outcome distribution is also valuable because it enables uncertainty estimation in the target domain, potentially helping practitioners assess the reliability of individual predictions.
Critical weaknesses

1 The empirical results show that IWCAE outperforms the evaluated baselines, but the necessity of each component is insufficiently justified. Although the paper motivates the use of CVAE, importance weighting, and the MMD penalty, their individual contributions to the reported improvements remain unclear. Controlled experiments isolating the effects of these components would help establish which elements are responsible for the gains.

  1. The connection between assumption and training objective is not clear to me. The conditional likelihood objective encourage the model to preserves source-domain predictive information, while MMD encourages marginal alignment. Conditional invariance across domains remains an assumption. It would be helpful to clarify under what conditions the conditional invariance assumption hold or how to enforce it.
Questions for authors

Clarification In the experiments, is It CVAE+MMD equivalent to IWCAE with K =1? Also, how was K=5 selected, and have the authors tried different K and compare the effects?

Clarification Could the authors clarify why SIR is needed when predictive samples can be generated directly by sampling from the Gaussian conditional prior and then decoding? What advantage does SIR offer in this setting?

Suggestion for future work One of the main motivations of this work is to learn a predictive distribution. It would be insightful to compare IWCAE with a deterministic predictor counterpart using the same backbone, representation mapping, and MMD alignment to isolate the benefit of distributional modeling. Evaluating both predictive accuracy and the practical utility of the uncertainty estimates would help strengthen this motivation.

Minor comments:

  • Typo on Line 206, k=1k = 1.
  • Figure 4 is a bit hard to read. The x labels in 4b are misaligned.
4 comments

We sincerely thank the reviewer for the constructive comments and valuable suggestions. We provide point-by-point responses to the raised concerns and questions below.

Response to Critical Weaknesses

1: To clarify the contributions of the individual components of IWCAE, we consider four model configurations: vanilla CVAE, IWCAE with K=1K=1, IWCAE with K=5K=5 without representation alignment, and the full IWCAE with K=5K=5. The configurations are summarized in the following table.

ConfigurationKKRepresentationMMD alignmentImportance weighting
Vanilla CVAE1τ(X)=X\tau(X)=XNoNo
IWCAE (K=1K=1)1LearnedYesNo
IWCAE (No alignment)5τ(X)=X\tau(X)=XNoYes
Full IWCAE5LearnedYesYes

Vanilla CVAE uses a single-sample variational objective (K=1K=1) without representation alignment, where τ(X)=X\tau(X)=X. Comparing IWCAE with K=1K=1 and K=5K=5 isolates the contribution of importance weighting, while comparing IWCAE with and without representation alignment at K=5K=5 evaluates the contribution of invariant representation learning and MMD alignment. We have added the ablation results for all three tasks to Table 3 in Section 5.4 of the main text. Overall, the full IWCAE framework achieves the best performance across all evaluated OOD settings. Together, these experiments provide a more systematic evaluation of the roles of these components in the proposed framework.

2: We thank the reviewer for raising this important question. We agree that the conditional invariance PY∣τ(X)tr=PY∣τ(X)teP^{\mathrm{tr}}_{Y\mid \tau(X)}=P^{\mathrm{te}}_{Y\mid \tau(X)} is an assumption and, in general, cannot be directly verified without labeled target-domain data. However, some form of conditional invariance between the training and test distributions is necessary. Without such an assumption, there is no restriction on how the relationship between the covariates and the outcome may change from training to testing, and therefore no principled way to transfer the predictive relationship learned from the training data to the test distribution.

The general conditional invariance assumption assumes PY∣Xtr=PY∣XteP^{\mathrm{tr}}_{Y\mid X}=P^{\mathrm{te}}_{Y\mid X}, so that the conditional relationship learned from the training distribution remains valid under the test distribution, regardless of whether there is a distribution shift in the covariates. Our assumption includes this setting as a special case by letting τ\tau be the identity function. In this sense, our conditional invariance assumption is weaker than the general conditional invariance assumption, as it only requires the conditional relationship to remain invariant through τ(X)\tau(X), a reduced form of XX, rather than through the entire covariate vector XX. In fact, if no form of conditional invariance is assumed and the marginal distributions of XX differ between the training and test samples, then the joint distributions of (Y,X)(Y,X) could entirely different across the two domains. In this case, there is no principled justification for using the training data to learn a predictive model for outcomes in the test sample.

One caveat is that τ(X)\tau(X) needs to retain sufficient information for predicting YY. This property can be assessed directly using the training sample by evaluating the cross-validated prediction accuracy. If the predictive performance for YY given τ(X)\tau(X) is comparable to that for YY given XX, then little predictive information is lost by restricting attention to the potentially lower-dimensional representation τ(X)\tau(X).

Response to Questions

1: CVAE+MMD is not equivalent to IWCAE with K=1K=1. Specifically, CVAE+MMD combines a vanilla CVAE, without importance-weighted learning or invariant representation learning, with MMD-based alignment between the estimated predictive distributions P^θ^tr(⋅∣X)\hat{P}^{\mathrm{tr}}_{\widehat{\theta}}(\cdot \mid X) and P^θ^te(⋅∣X)\hat{P}^{\mathrm{te}}_{\widehat{\theta}}(\cdot \mid X). The comparison shows that IWCAE achieves better performance than CVAE+MMD.

To investigate the effect of the number of importance samples KK, we conducted an additional sensitivity analysis with K∈1,3,5,7,10K \in \\{ 1,3,5,7,10 \\} in the dementia classification experiment. The results are summarized in Table 4 at Section A.3.1. The results show that IWCAE generally outperforms the baseline methods across different choices of KK, demonstrating its robustness to this hyperparameter. Since increasing KK introduces additional computational costs, we select K=5K=5 as a reasonable balance between predictive performance and computational efficiency.

Response to Questions

2: Our motivation for introducing SIR follows established proposal-based inference approaches in deep latent-variable models (Sohn et al., 2015; Mattei and Frellsen, 2019). In particular, Sohn et al. (2015) demonstrated that importance sampling can improve the efficiency of conditional likelihood estimation, while Mattei and Frellsen (2019) employed SIR to generate samples from conditional distributions in latent-variable models. An important advantage of proposal-based inference is its flexibility in accommodating complex conditional prior distributions for which direct sampling may be difficult but density evaluation remains feasible. Although direct sampling from the prior is feasible for the Gaussian prior considered in our experiments, SIR provides a more general inference strategy that can accommodate non-Gaussian prior distributions. Furthermore, as we aim to develop IWCAE into a modular framework applicable to different backbone models, adopting SIR as an optional inference procedure offers additional flexibility. We have clarified this motivation and the role of SIR in Section 3.4.

3: We thank the reviewer for this insightful suggestion. We focus on estimating the conditional distribution, rather than making a single deterministic point prediction, because the conditional distribution provides a natural way to quantify prediction uncertainty. Quantifying prediction uncertainty is particularly important in medical applications. For example, tumor segmentation results may vary across scanners, imaging protocols, or acquisition settings. Point estimates alone may make it difficult to determine whether an observed change in tumor size reflects a true biological change or simply variability introduced by the imaging process.

In an ongoing related study on image segmentation in our lab, we further compare IWCAE with its deterministic baseline. Our preliminary results suggest that, with appropriate hyperparameter tuning, IWCAE can maintain point predictive accuracy. The proposed framework also provides predictive uncertainty estimates, allowing us to assess confidence in the predictions.

4: Thank you for pointing out these issues. We have corrected the typo on Line 206, where k=1k=1 should be K=1K=1. We have also revised Figure 4 to improve its readability, particularly by rotating the x-axis labels in Figure 4(b) to 90 degrees to ensure proper alignment.

Thank you for the detailed rebuttal and for clarifying my questions. I found the clarification on when τ(X)\tau(X) is used versus the raw data XX helpful and would recommend add it to the paper. I am raising my score from 2 to 3.

Thank you very much for your positive feedback and for reconsidering our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.