On the Nature of Domain Generalization

Domain generalization essentially faces the challenge where distribution shift leads to a generalization gap between the source and target domains, while any information on the target domains is unavailable during training. In this work, by introducing representation, we reveal that matching representation marginals and learning invariant features are equally important and both necessary for narrowing the generalization gap. In particular, we show that marginal matching can be regarded as a prerequisite for invariance learning. We then uncover a “domino effect” in marginal matching under a mild assumption, which guarantees that, without requiring knowledge of any target marginals, matching source marginals alone is sufficient to address the prerequisite. Additionally, we analyze the generalization discrepancy between the minimizer of the empirical risk subject to marginal matching and invariance constraints and the optimal minimizers of the expected risk, showing that domain generalization methods have no theoretical advantage in reducing this generalization discrepancy compared with Empirical Risk Minimization. This suggests that research focusing merely on algorithmic-level improvement cannot lead to real breakthroughs.