acceptodds
Under review as a conference paper at ICLR 2027

Testing the domain invariance hypothesis in low target data Domain Adaptation

Abstract

Out-of-Distribution (OOD) generalization aims to ensure robust model performance when a target distribution differs from the source distribution observed during training. Within this broader problem, Domain Generalization (DG) considers the challenging setting where no target-domain data are available during training, whereas Domain Adaptation (DA) leverages target-domain information to facilitate generalization to the target domain. In practice, the transition between DG and DA depends on the amount of target data and is particularly relevant when only a few target samples are available. We therefore revisit this dichotomy by introducing a unified formulation that treats DG and DA as continuum governed by the amount of available target data. To this end, we extend the experimental protocol DomainBed designed for DG experiments to Unsupervised Domain Adaptation (UDA) and Supervised Domain Adaptation (SDA), and empirically investigate how adaptation occurs through domain discrepancy and decision boundaries evolutions. We investigate the role of the domain-invariant hypothesis across the transition from DG to DA. If domain invariance contributes to generalization, greater domain invariance should be systematically associated with improved target-domain performance. We therefore compare statistical (MMD) and adversarial (DANN) alignment with alignment-free ERM under varying target-data availability, allowing us to examine whether alignment systematically translates into improved target performance along the proposed critical dimension. To assess the generality of our findings across modalities, we conduct experiments on the CWWS speech dataset, which includes various types of corruptions, and the popular VLCS image dataset. Across both data modalities, we observe comparable overall trends, while the supervised and unsupervised adaptation continuums exhibit distinct behaviors as target-data availability increases. Explicit domain-alignment does not outperform classical ERM in DGSDA while they reach early performance plateaus in DGUDA despite target error reduction in all scenarios. Quantitative analysis identifies the factors that account for these differences and assesses the empirical validity of the domain-invariant hypothesis in our experimental setting. The code will be publicly released upon publication of the paper.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Domain generalization, Supervised Domain Adaptation, Unsupervised Domain Adaptation, Low target data availability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8hpPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors were trying to investigate whether forced feature alignment actually helps machine learning models adapt when target-domain data is extremely scarce.

Critical strengths
  • the paper is easy to follow.
Critical weaknesses
  • The authors claim that their findings challenge the Domain-Invariance Hypothesis in low-target-data regimes. However, this failure is a well-known property of statistical learning theory regarding finite-samples. Under extreme asymmetry (nS≫nTn_S \gg n_T), the variance of empirical metrics (e.g., MMD or HΔH\mathcal{H}\Delta\mathcal{H}-divergence) is proportional to O(1/nT)O(1/\sqrt{n_T}). The performance degradation in low-data regimes is not a failure of domain invariance, but the failure of optimizing high-variance empirical objectives under finite target samples. The paper misattributes sample-complexity bounds to the limitation of domain invariance itself.
  • Regarding the joint error, it has been theoretically discussed in "On Learning Invariant Representation for Domain Adaptation", ICML 2019; Zhao et al. proved that when marginal class distributions differ across domains or when conditional distributions shift, forcing the domain-invariant representations leads to a lower bound on the joint error. Mathematically, as feature discrepancy decrease, the optimal joint error will increase.
Questions for authors

see weakness.

1 comment

We thank the reviewers for their helpful discussion and constructive comments. We address the points below.

  1. We agree with the reviewer that optimizing empirical objectives under limited target-domain samples is subject to finite-sample effects, and that empirical discrepancy estimators can exhibit increased variance as the number of target samples decreases. However, we believe that this explanation alone does not fully account for the observations reported in our experiments.

First, an important distinction must be made between the number of target samples used during training and the number of samples used to estimate representation-level discrepancies. In our experiments, for instance Figure 3, discrepancies between domains are computed in the learned embedding space using held-out test sets, whose sample sizes remain fixed across all target-data granularities (2%, 10%, 50%, and 100%). Therefore, the observed differences in discrepancy cannot be attributed to a reduction in the sample size used to estimate these quantities. Instead, they reflect changes in the learned representations resulting from training with different amounts of target-domain supervision.

Second, as illustrated in Figure 3 (UDA - MMD), we observe a reduction in source-target discrepancy at 2% target data, which remains approximately unchanged when increasing the amount of target data up to 100%. We observe consistent trends on the CWWS dataset (Figure 6). These results suggest that the optimization of the MMD loss is effective despite the high variance of the objective under finite target samples pointed by the reviewer. Actually we observe a quick saturation of UDA performance with the quantity of target data, suggesting that the main limitation is not an optimization problem with few target data but the effective use of more target data.

  1. In "On Learning Invariant Representation for Domain Adaptation", ICML 2019; Zhao et al. show that when feature discrepancy decreases, the optimal joint error can increase (Theorem 4.3). However, our paper is in the particular setting of covariate shift, where this result does not hold (D_JS(D_S^Y , D_T^Y )=0) .

In the setting of covariate shift, Theorem 4.1 of Zhao et al. says that ‘to guarantee successful domain adaptation, it suffices to match the marginal distributions while achieving small error on the source domain’. In our experimental results, we reduce feature discrepancy (measured by MMD) and keep source error small (measured by epsilon) without reducing substantially the target error. Our experimental results constitute a test of this theory of Zhao et al in the specific setting of covariate shift. In our implementation, this invariance principle is ineffective. We will make clearer in the manuscript that our experimental results in the case of covariate shift are in contradiction with the theory.

Reviewer_PqwuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the domain invariance hypothesis in low-target-data domain adaptation by studying the transition from Domain Generalization (DG) to Unsupervised and Supervised Domain Adaptation (UDA/SDA). The authors extend DomainBed to evaluate ERM, MMD, and DANN under varying amounts of target data. Experiments on visual and speech datasets show that UDA performance tends to saturate early, while SDA benefits more from additional target data. The results also suggest that stronger domain alignment does not necessarily lead to better target-domain performance.

Critical strengths
  1. The paper provides a systematic evaluation of the transition from DG to DA under varying amounts of target data. This is an interesting research perspective that helps better understand how target data availability affects domain adaptation and generalization.

  2. The paper is well organized and easy to follow, with a clearly described experimental setup. The evaluation covers both supervised and unsupervised adaptation across visual and speech tasks, providing useful comparisons under different target-data settings.

Critical weaknesses
  1. Limited experimental scope. Since the paper focuses on systematic empirical analysis rather than proposing a new method, a more comprehensive evaluation is expected. However, the experiments only cover three algorithms (ERM, MMD, and DANN) and two datasets, which is rather limited compared with established benchmarks such as DomainBed. This makes it difficult to draw general conclusions about the domain invariance hypothesis.

  2. Insufficient experimental reliability. DA/DG experiments are often highly sensitive to random seeds, yet the paper does not report mean and standard deviation across multiple independent runs. Many conclusions are drawn from average trends across different target-data fractions, without sufficient analysis of sampling variance or statistical significance. Moreover, hyperparameter search is conducted with only a single seed, further raising concerns about the reliability of the experimental findings.

  3. The main experiments use target-domain validation for model selection, even in DG and UDA settings where labeled target data should not be available. Although source-based validation is discussed in the appendix, the main conclusions still rely on oracle selection. This raises concerns about the fairness of the evaluation and whether the findings hold under more realistic settings.

Questions for authors
  1. The experiments only cover three algorithms and two datasets. Could the authors provide additional evaluations on more widely used DG/DA benchmarks, such as PACS, OfficeHome, or DomainNet, to better support the generality of their conclusions?

  2. Given the sensitivity of DG/DA experiments to random seeds, could the authors report mean and standard deviation across multiple independent runs? Also, how stable are the conclusions under different hyperparameter search seeds and target-data sampling?

  3. The main experiments rely on oracle model selection using labeled target-domain validation data. Could the authors clarify whether the main conclusions, particularly regarding the relationship between domain invariance and target performance, still hold under source-based model selection?

4 comments

We thank the reviewers for their helpful discussion and constructive comments. We address the points below.

Question 1 - Limited experimental scope: We acknowledge that a broader evaluation, involving more algorithms and datasets as in benchmarks such as DomainBed could provide further insights. However, our objective was not to establish an exhaustive benchmark of domain generalization methods, but rather to conduct a systematic empirical investigation of the domain invariance hypothesis across different adaptation settings, with a particular focus on the low-target-data regime. We therefore selected ERM as a baseline and MMD and DANN as representative approaches relying on distribution alignment, allowing us to investigate whether and under which conditions invariance-based approaches provide an advantage over ERM. Regarding the datasets, we considered two distinct modalities (image and audio) that exhibit remarkably similar behavioral trends. Furthermore, our experimental study involved 800 independent training runs across domain generalization, supervised domain adaptation, and unsupervised domain adaptation. Extending this evaluation to additional datasets would substantially increase the computational cost. We insist on the fact that our experimental coverage is narrower in terms of datasets compared to published DG or DA benchmarks, but much deeper in terms of scenarios: DG, UDA and SDA with various amounts of target data availability.

As mentioned in the submitted manuscript, we acknowledge that our conclusions should be interpreted within the scope of the considered algorithms and datasets. Nevertheless, the consistency of the observed trends across modalities and adaptation settings provides encouraging evidence supporting our empirical findings.

Question 2 - Insufficient experimental reliability. We would like to clarify our experimental protocol regarding random seeds. Following Yu et al. (2024), we perform hyperparameter selection using a single random seed (trial_seed = 0). Once the best hyperparameter configuration has been identified through grid search, we fix this configuration and evaluate its performance across three independent training runs, using trial_seed = 0, 1, 2. We provide in the following Table mean and standard deviation of the performance metrics for these three seeds for the MMD algorithm.

The additional MMD experiments provide insights into both training variability and the effect of target-data availability. This indicates relatively limited variability across training runs for the selected hyperparameter configurations. Importantly, the magnitude of this variability is generally smaller than the performance differences observed across target-data-availability settings. For instance, in the SDA setting, the target error decreases from 0.266 to 0.172 on VLCS and from 0.151 to 0.095 on CWWS when moving from DG to DA100%. These improvements are substantially larger than the corresponding standard deviations.

Due to computational constraints during the rebuttal period, we were unable to complete the corresponding multi-seed experiments for ERM and DANN. However, we will extend the same evaluation protocol to both algorithms in the final manuscript: hyperparameters will be selected using trial_seed = 0, and the selected configurations will then be evaluated across three training seeds (trial_seed = 0, 1, 2). We will report the corresponding mean and standard deviation for both source and target errors, providing a consistent assessment of training variability across ERM, MMD, and DANN.

\hline \text{Dataset} & \text{Mode} & \text{Enrichment} & \text{Target Error} & \text{Source Error} \\\\ \hline \text{VLCS} & \text{UDA} & \text{DG} & 0.266 \pm 0.012 & 0.194 \pm 0.006 \\\\ \text{VLCS} & \text{UDA} & \text{DA2\\%} & 0.238 \pm 0.009 & 0.185 \pm 0.006 \\\\ \text{VLCS} & \text{UDA} & \text{DA100\\%} & 0.238 \pm 0.005 & 0.185 \pm 0.007 \\\\ \text{VLCS} & \text{SDA} & \text{DG} & 0.266 \pm 0.012 & 0.194 \pm 0.006 \\\\ \text{VLCS} & \text{SDA} & \text{DA2\\%} & 0.223 \pm 0.004 & 0.190 \pm 0.007 \\\\ \text{VLCS} & \text{SDA} & \text{DA100\\%} & 0.172 \pm 0.008 & 0.187 \pm 0.003 \\\\ \hline \text{CWWS} & \text{UDA} & \text{DG} & 0.151 \pm 0.011 & 0.108 \pm 0.009 \\\\ \text{CWWS} & \text{UDA} & \text{DA2\\%} & 0.136 \pm 0.008 & 0.106 \pm 0.007 \\\\ \text{CWWS} & \text{UDA} & \text{DA100\\%} & 0.120 \pm 0.004 & 0.100 \pm 0.008 \\\\ \text{CWWS} & \text{SDA} & \text{DG} & 0.151 \pm 0.011 & 0.108 \pm 0.009 \\\\ \text{CWWS} & \text{SDA} & \text{DA2\\%} & 0.132 \pm 0.006 & 0.106 \pm 0.007 \\\\ \text{CWWS} & \text{SDA} & \text{DA100\\%} & 0.095 \pm 0.010 & 0.096 \pm 0.010 \\\\ \hline \end{array}$$

Question 3 - Model selection method:

We thank the reviewer for raising this important point. To investigate this issue, Figure 5 compares oracle and source-based model selection on both datasets along the DG->UDA continuum. We observe no major differences between the two strategies, apart from the expected slight performance improvement with oracle selection. Importantly, the main trends and conclusions, including the relationship between domain invariance and target performance, remain consistent.

We do not intend to downplay the importance of model selection. Rather, we use oracle selection in the subsequent analyses to control for variations introduced by different selection strategies and better disentangle the phenomena associated with the DG-to-SDA and DG-to-UDA continua. This allows us to focus on the effects of the different regimes without introducing model selection as an additional confounding factor.

The conclusions about target performance still hold with the source-based selection method as shown by Figure 5. In addition, we reproduce Figure 6 for UDA with source-based selection method. It shows that the relationship between domain invariance and target performance still holds. The heatmap with source-validation in UDA continuum is now available in the supplementary materials and we will be added in the revised manuscript.

Thank you for the clarifications. Two core questions remain:

  1. Section 3.2 selects the best model using at most the available target-data budget. Could the observed UDA plateau partly reflect repeatedly selecting the same lower-budget model, rather than saturation when training with more target data? How are these explanations distinguished?

  2. The additional multi-seed results concern MMD alone, while Figure 5 shows that model selection changes its ranking relative to ERM on VLCS. What evidence supports the robustness of the cross-algorithm conclusions to both random variation and model-selection strategy?

We thank the reviewer for the insightful feedback.

  1. Indeed the UDA plateau corresponds to repeatedly selecting the same model. We add in the supplementary material two additional plots. First, we plot for UDA settings the performance of the best model using exactly the available target-data budget with “*” markers and the selected experiment by our method with “o” markers. It shows that the best model does not always correspond to the largest target data budget and in some cases reduces the performance. We also provide an additional plot showing which model is selected with our proposed selection strategy for each target-data budget compared to the maximum available target enrichment in dark dotted lines. We observe similar trends between the two alignment algorithms MMD and DANN in terms of integration abilities.

These plots support the conclusion of a saturation of UDA with more target data. Adding more target data does not improve performance and can even degrade performance. This leads to the selection of a model with less target-data budget with our strategy. We thank the reviewer for this discussion and we will add these additional comments in the final manuscript.

  1. We do not claim that our results support a target performance difference between the three algorithms. Instead we conclude that the three considered algorithms exhibit comparable behavior across the target data continuums, suggesting that the learned representations do not systematically capture the domain structure required for more effective target-domain integration'. We extend the results from a large body of works from DomainBed original paper (Gulrajani & Lopez-Paz, 2021) defending that domain generalization algorithms only provide limited gains among ERM. Since ERM is not usable in UDA, we show that these domain generalization/adaptation algorithms have also limited potential in the UDA continuum.

We are unable to run the multi-seeds experiment for all configurations during the discussion period. However we will add multi-seed results in the final version of the manuscript. The standard deviation of the target error for MMD (for three seeds) is of the order of 1%. In Figure 5 with UDA, a standard deviation of this order of magnitude would not support a performance difference between the algorithms for CWWS (target and source validation), and for VLCS (target validation). For VLCS source validation, there could be a significant difference in performance between MMD (for 100%, average target error=0.25) and the two other algorithms (0.22 for ERM, 0.23 for DANN).

Reviewer_jqEKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers the setting of limited Target data for domain adaptation. They connect domain generalization (no target data available) to unsupervised and supervised DA. They extend the evaluation protocol on DomainBed.

Critical strengths

a. Strong empirical analysis on a common framework for UDA and DG. Strong experiments with different modalities.

Critical weaknesses

a. Limited novelty: The main finding that domain alignment does not necessarily improve target performance is there in previous work (Zhao et al., 2019 - https://proceedings.mlr.press/v97/zhao19a.html). Likewise, early saturation in UDA has been discussed in Kalluri et al. 2024 (https://mlanthology.org/eccv/2024/kalluri2024eccv-udabench/). The paper's contributions beyond these findings are not sufficiently clear. Both papers are cited, but not sufficiently addressed in terms of novelty.

b. Limited experimentation: The evaluation covers only two datasets and three algorithms, without reported variance across multiple runs. Is the proposed method generalizable and reliable?

Questions for authors

a. Joint-error computation: How is the empirical optimal joint-error term \lambda_ST computed? Is a separate classifier trained on the learned representations? b. Lack of Variance Results: Are the performance differences between ERM, MMD, and DANN consistent across multiple random seeds?

2 comments

We thank the reviewer for the helpful discussion and constructive comments. We address the points below.

Weakness a. . Limited novelty: We recall that the main novelty and contribution of this work is the introduction of a methodology to evaluate the benefit of invariance when introducing a target domain. We agree with the reviewer that we should insist more clearly on how our research contributes from the existing literature.

Subsection 5.1 shows the observed phenomenon without claiming novelties but is essential for situating the findings within the broader framework of existing research. Indeed, we mention in the related works and in Section 5.1 that our experimental setup leads to consistent observations with UDA-Bench (Kalluri et al., 2024) which we never claim as novel but only as empirical confirmation. Also, our experimental protocol based on DomainBed controlled study across algorithms and datasets empirically confirms the validity of this claim.

The main novelties of our paper are presented in Subsection 5.2. Regarding the analysis of Zhao et al., 2019, we would like to be precise about their claim and our claim. First, in On Learning Invariant Representations for Domain Adaptation by Zhao et al., the authors derive in Theorem 4.1 an extension of the well-known Ben-David upper bound highlighting a main difference in the third term that should vanish when considering covariate shift. In our work we only consider covariate shifts in both image and audio modalities. In this setting, ‘it suffices to match the marginal distributions while achieving small error on the source domain’ according to Zhao et al, 2019. Surprisingly, our experimental results (Figure 3 and Figure 6) show that the empirical Joint Error is not vanishing but only remaining constant (CWWS dataset) and increasing (VLCS dataset) along the target data availability dimension. This is a new analysis in covariate shift studies.

Weakness b. Limited experimentation: Regarding the choice of algorithms, our objective was not to provide an exhaustive benchmark, but rather to investigate how invariance can be challenged in supervised and unsupervised domain adaptation, particularly in the low-target-data regime. We therefore selected three (two + ERM) representative algorithms commonly used in unsupervised domain adaptation to evaluate their performance under limited target data and compare them with the standard ERM baseline in terms of performance and representation stability.

Regarding the choice of datasets, we agree that extending the evaluation to more datasets could provide stronger evidence of generalizability. However, the computational cost of our experimental study is substantial. Specifically, our evaluation involved: Domain Generalization (DG): 2 datasets × 20 hyperparameter configurations × 1 seed × 4 target-domain rollouts = 160 runs. Supervised Domain Adaptation (SDA): 2 datasets × 10 hyperparameter configurations × 1 seed × 4 target-domain rollouts × 4 target-data granularities (2%, 10%, 50%, and 100%) = 320 runs. Unsupervised Domain Adaptation (UDA): 2 datasets × 10 hyperparameter configurations × 1 seed × 4 target-domain rollouts × 4 target-data granularities (2%, 10%, 50%, and 100%) = 320 runs.

This amounts to 800 independent training runs across two datasets and three experimental settings. Despite these computational constraints, we considered two distinct modalities with 4 target domains each (8 in total) that exhibit remarkably similar behavioral trends, providing encouraging evidence of the generalizability of our findings. We nevertheless acknowledge that additional datasets and multiple random seeds would further strengthen the assessment of the method's generalizability and reliability.

In this section we answer the questions:

Question a: Joint-error computation: The empirical optimal joint-error term (λST\lambda_{ST}) is estimated by training a separate linear classifier on the learned representations (embeddings) of both the source and target domains. Specifically, we combine the source and target embeddings and train the classifier to distinguish between the different classes. The resulting test classification error provides an empirical estimate of the joint-error term (λST\lambda_{ST}). We will clarify this computation and provide further details in the revised manuscript.

Question b: Lack of Variance Results: According to previous works [In search of lost domain generalization. Ishaan Gulrajani and David Lopez-Paz. 2021] we show that there is no consistent difference between the three algorithms in terms of performance. We extend this result from DG to the proposed target data availability dimension for UDA and SDA. Consistently with [Yu et al., 2024], we only use one random seed for the hyperparameter search. However [Yu et al., 2024] report the performance for the selected hyperparameters for three seeds.

To address the reviewer's concern, we provide additional experiments for MMD, reporting the mean and standard deviation of source and target errors across three random seeds. The results are presented in the following Table. They provide an estimate of the variability associated with MMD across datasets and target-data-availability settings. Overall, the standard deviations of the target errors range from 0.004 to 0.012, while those of the source errors range from 0.003 to 0.010. These results suggest that the observed performance differences across target-data-availability settings are generally larger than the variability across random seeds in several configurations.

Unfortunately, we were unable to complete the corresponding multi-seed experiments for ERM and DANN within the rebuttal period. We acknowledge this limitation and will include the corresponding mean and standard deviation results for all three algorithms in the final manuscript.

\hline \text{Dataset} & \text{Mode} & \text{Enrichment} & \text{Target Error} & \text{Source Error} \\\\ \hline \text{VLCS} & \text{UDA} & \text{DG} & 0.266 \pm 0.012 & 0.194 \pm 0.006 \\\\ \text{VLCS} & \text{UDA} & \text{DA2\\%} & 0.238 \pm 0.009 & 0.185 \pm 0.006 \\\\ \text{VLCS} & \text{UDA} & \text{DA100\\%} & 0.238 \pm 0.005 & 0.185 \pm 0.007 \\\\ \text{VLCS} & \text{SDA} & \text{DG} & 0.266 \pm 0.012 & 0.194 \pm 0.006 \\\\ \text{VLCS} & \text{SDA} & \text{DA2\\%} & 0.223 \pm 0.004 & 0.190 \pm 0.007 \\\\ \text{VLCS} & \text{SDA} & \text{DA100\\%} & 0.172 \pm 0.008 & 0.187 \pm 0.003 \\\\ \hline \text{CWWS} & \text{UDA} & \text{DG} & 0.151 \pm 0.011 & 0.108 \pm 0.009 \\\\ \text{CWWS} & \text{UDA} & \text{DA2\\%} & 0.136 \pm 0.008 & 0.106 \pm 0.007 \\\\ \text{CWWS} & \text{UDA} & \text{DA100\\%} & 0.120 \pm 0.004 & 0.100 \pm 0.008 \\\\ \text{CWWS} & \text{SDA} & \text{DG} & 0.151 \pm 0.011 & 0.108 \pm 0.009 \\\\ \text{CWWS} & \text{SDA} & \text{DA2\\%} & 0.132 \pm 0.006 & 0.106 \pm 0.007 \\\\ \text{CWWS} & \text{SDA} & \text{DA100\\%} & 0.095 \pm 0.010 & 0.096 \pm 0.010 \\\\ \hline \end{array}$$

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.