- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors were trying to investigate whether forced feature alignment actually helps machine learning models adapt when target-domain data is extremely scarce.
Critical strengths
- the paper is easy to follow.
Critical weaknesses
- The authors claim that their findings challenge the Domain-Invariance Hypothesis in low-target-data regimes. However, this failure is a well-known property of statistical learning theory regarding finite-samples. Under extreme asymmetry (), the variance of empirical metrics (e.g., MMD or -divergence) is proportional to . The performance degradation in low-data regimes is not a failure of domain invariance, but the failure of optimizing high-variance empirical objectives under finite target samples. The paper misattributes sample-complexity bounds to the limitation of domain invariance itself.
- Regarding the joint error, it has been theoretically discussed in "On Learning Invariant Representation for Domain Adaptation", ICML 2019; Zhao et al. proved that when marginal class distributions differ across domains or when conditional distributions shift, forcing the domain-invariant representations leads to a lower bound on the joint error. Mathematically, as feature discrepancy decrease, the optimal joint error will increase.
Questions for authors
see weakness.
We thank the reviewers for their helpful discussion and constructive comments. We address the points below.
First, an important distinction must be made between the number of target samples used during training and the number of samples used to estimate representation-level discrepancies. In our experiments, for instance Figure 3, discrepancies between domains are computed in the learned embedding space using held-out test sets, whose sample sizes remain fixed across all target-data granularities (2%, 10%, 50%, and 100%). Therefore, the observed differences in discrepancy cannot be attributed to a reduction in the sample size used to estimate these quantities. Instead, they reflect changes in the learned representations resulting from training with different amounts of target-domain supervision.
Second, as illustrated in Figure 3 (UDA - MMD), we observe a reduction in source-target discrepancy at 2% target data, which remains approximately unchanged when increasing the amount of target data up to 100%. We observe consistent trends on the CWWS dataset (Figure 6). These results suggest that the optimization of the MMD loss is effective despite the high variance of the objective under finite target samples pointed by the reviewer. Actually we observe a quick saturation of UDA performance with the quantity of target data, suggesting that the main limitation is not an optimization problem with few target data but the effective use of more target data.
In the setting of covariate shift, Theorem 4.1 of Zhao et al. says that ‘to guarantee successful domain adaptation, it suffices to match the marginal distributions while achieving small error on the source domain’. In our experimental results, we reduce feature discrepancy (measured by MMD) and keep source error small (measured by epsilon) without reducing substantially the target error. Our experimental results constitute a test of this theory of Zhao et al in the specific setting of covariate shift. In our implementation, this invariance principle is ineffective. We will make clearer in the manuscript that our experimental results in the case of covariate shift are in contradiction with the theory.