Domain generalization essentially faces the challenge where distribution shift leads to a generalization gap between the source and target domains, while any information on the target domains is unavailable during training. In this work, by introducing representation, we reveal that matching representation marginals and learning invariant features are equally important and both necessary for narrowing the generalization gap. In particular, we show that marginal matching can be regarded as a prerequisite for invariance learning. We then uncover a “domino effect” in marginal matching under a mild assumption, which guarantees that, without requiring knowledge of any target marginals, matching source marginals alone is sufficient to address the prerequisite. Additionally, we analyze the generalization discrepancy between the minimizer of the empirical risk subject to marginal matching and invariance constraints and the optimal minimizers of the expected risk, showing that domain generalization methods have no theoretical advantage in reducing this generalization discrepancy compared with Empirical Risk Minimization. This suggests that research focusing merely on algorithmic-level improvement cannot lead to real breakthroughs.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.Reviewer_sr91OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
This paper studies domain generalization from an information-theoretic perspective. It decomposes the generalization gap into marginal and conditional distribution discrepancies, arguing that marginal matching and invariance learning are both necessary. The main theoretical contribution is a "domino effect" (Theorem 4), claiming that matching source representations to a reference distribution also reduces source-target discrepancy. The paper further discusses the limitations of DG methods compared with ERM. The claims are illustrated using synthetic data and Colored MNIST.
Critical strengths
The paper tries to explain why marginal matching helps in DG, rather than just proposing another algorithm. The discussion connecting marginal matching, conditional invariance, and ERM is interesting and potentially useful.
Critical weaknesses
I have serious concerns about Theorem 4, which seems to be the main novelty of the paper.
The proof uses the triangle inequality to obtain
d(Pt,Pi)→d(Pt,p)
when d(Pi,p)→0. However, this does not imply d(Pt,Pi)→0 unless d(Pt,p)→0 is also guaranteed. I do not see how Assumption 1 provides this guarantee.
For example, consider Xs∼U[0,1], Xt=2, and p=δ0. Define
gn(x)=1x∈[0,1/n] or x=2.
Then
Psgn=(1−1/n)δ0+(1/n)δ1→p,
while Ptgn=δ1. Therefore,
\to\sqrt{\ln 2}\neq 0.$$
These are measurable representations in a countable function class, consistent with the approximability interpretation given by the authors. This seems to contradict the claimed domino effect. Also, the quantifier order in Assumption 1 is unclear and appears stronger than the separability condition described in the text.
2. Some theoretical conclusions seem stronger than what is actually proved.
The decomposition of the generalization bound does not establish that both marginal matching and invariance learning are necessary for good generalization. Similarly, showing that the DG and ERM upper bounds cannot be ordered does not prove that DG methods have no theoretical advantage over ERM. These claims also overlap with existing work on invariance and the limitations of DG. The experiments are relatively limited and do not resolve these theoretical concerns.
Questions for authors
Could the authors clarify the quantifiers in Assumption 1 and explain how it implies d(Pt,p)=d(Pi,p) in the proof of Theorem 4? How would the authors address the counterexample above?
Can the authors clarify whether the claims about the necessity of marginal matching and the lack of advantage over ERM are formal theoretical results, or only observations about the derived upper bounds?
This paper provides a theoretical analysis of domain generalization through representation learning. It decomposes the source-to-target generalization gap into discrepancies between representation marginals and conditional label distributions, arguing that marginal matching and invariant feature learning are both necessary. The paper further proposes a “domino effect,” claiming that matching source-domain representation marginals to a fixed distribution can also reduce source-target marginal discrepancies without access to target data. Finally, it analyzes the discrepancy between constrained domain-generalization solutions and optimal expected-risk minimizers, concluding that existing DG methods have no guaranteed theoretical advantage over ERM without additional assumptions about the data and optimal hypotheses. The theoretical claims are illustrated using synthetic experiments and a reduced Colored MNIST setting.
Critical strengths
The paper attempts to clarify the respective roles of marginal matching and invariant feature learning in domain generalization through a unified theoretical analysis. In particular, the decomposition of the generalization gap provides a useful conceptual framework for understanding why these objectives may be complementary.
Critical weaknesses
The experimental section is too limited to support the paper’s broad conclusions. Synthetic experiments can illustrate the proposed theorem, but they do not establish that the claimed limitations of DG algorithms manifest in realistic settings. The only semi-synthetic experiment uses 10% of Colored MNIST, which the authors acknowledge may yield noisy accuracy estimates. No established real-world DG benchmark or standardized model-selection protocol is included.
Questions for authors
Q1. Can the authors validate the theoretical conclusions on established real-world DG benchmarks such as PACS, VLCS, OfficeHome, TerraIncognita, or DomainNet under a standardized evaluation protocol?
Q2. Why does Table 2 emphasize an average over source and target domains, rather than target-domain performance as the primary DG metric?
Q3. Were hyperparameters selected exclusively using source-domain validation data? Please clarify whether any target-domain results were consulted during model or hyperparameter selection.
Q4. Do the conclusions remain stable when the full Colored MNIST dataset is used, given that the current experiment uses only 10% of the data and is acknowledged to produce potentially noisy accuracy estimates?
Q5. Can the authors compare ERM and the proposed combinations with a broader set of representative DG methods to determine whether the theoretical limitation is reflected empirically?
This paper investigates domain generalization (DG) from an information-theoretic perspective, focusing on the relationship between representation marginal matching, invariance learning, and generalization performance on unseen target domains. The authors first derive a generalization bound based on the Jensen–Shannon divergence between source and target joint distributions, and further decompose this discrepancy into marginal and conditional components. Based on this analysis, they argue that marginal matching and invariance learning are both necessary for DG.
The paper's central theoretical claim is a so-called "domino effect": under an approximability assumption on the representation function class, matching source-domain representation marginals to a reference distribution is claimed to reduce discrepancies involving unseen target domains. The authors also derive bounds comparing regularized empirical-risk minimizers with domain-specific optimal predictors, arguing that these bounds do not establish a theoretical advantage of DG methods over standard ERM.
The empirical evaluation includes synthetic Gaussian data experiments using several distribution-matching methods and Colored MNIST experiments combining marginal matching with invariant risk minimization. While the experiments report improvements in distribution alignment and target-domain accuracy, the extent to which the theoretical and empirical results support the paper's central claims remains questionable.
Critical strengths
The paper addresses a fundamental question in domain generalization: why should aligning representations across observed source domains improve generalization to unseen target domains? The proposed information-theoretic perspective, which separates marginal and conditional distribution discrepancies, provides a potentially useful framework for investigating this question. A rigorous explanation of when source-domain alignment can guarantee target-domain generalization would be a meaningful contribution to the field.
Critical weaknesses
The paper contains serious flaws in its theoretical analysis, particularly in the two results central to its claimed contributions.
1. The central "domino effect" in Theorem 4 is not established under a meaningful set of assumptions.
Assumption 1 requires the existence of a single sequence (gk) whose induced distributions converge to those of everyg∈G. By uniqueness of limits, this forces all representations in G to induce the same distribution under any fixed input distribution, making the claimed convergence toward a distinct reference distribution vacuous.
Moreover, the proof in Appendix A.3 does not justify the crucial equality d(PT,p)=d(Pi,p), since Assumption 1 provides no relationship between source and target distributions. Under the conventional interpretation of approximability, the claimed implication is also not generally valid.
As Theorem 4 is explicitly presented as the paper's most important finding, this issue fundamentally undermines its central theoretical contribution.
2. Corollary 2 does not provide a valid generalization guarantee for the learned predictor.
The analysis applies a Hoeffding concentration bound for fixed hypotheses to the data-dependent minimizer h=argminhRλ(h), without establishing a uniform generalization bound or imposing appropriate complexity or stability assumptions.
This is not merely a missing technical detail: with an unrestricted class of measurable predictors, an empirical-risk minimizer can interpolate all training samples while having large target risk, even when the source and target representation distributions are identical and the invariance conditions hold.
Consequently, the claimed bound in Corollary 2 is not valid under the stated assumptions, and the subsequent comparison between DG methods and ERM lacks a sound theoretical foundation.
Questions for authors
Q1. Assumption 1 and the non-vacuity of Theorem 4. Assumption 1 states that ∃(gk)⊂G,∀g∈G,∀X∈X:Pgk(X)→Pg(X). For any fixed input distribution PX, uniqueness of limits implies Pg1(X)=Pg2(X) for all g1,g2∈G. Thus, the source representation distribution cannot change as g varies within G. For a fixed p=Pig(X), the premise dJS(Pig(X),p)→0 is therefore unattainable along any sequence in G. Is this quantifier order intended? If so, can you provide a non-vacuous example satisfying the premise of Theorem 4?
Q2. The unjustified equality in the proof of Theorem 4. In Appendix A.3, Assumption 1 is used to conclude dJS(Ptg(X),p)=dJS(Pig(X),p). However, even if Assumption 1 applies to both domains, it only gives Pigk(X)→Pig(X) and Ptgk(X)→Ptg(X), which does not establish this equality. Indeed, take G=g with g(x)=x, Xi≡0, Xt≡1, and p=43δ0+41δ1. Assumption 1 holds with gk=g, but dJS(Pig(X),p)=dJS(Ptg(X),p). How is the equality justified? Furthermore, if the assumed convergence is weak convergence, it does not imply convergence in JS distance: δ1/k⇒δ0, whereas dJS(δ1/k,δ0)=ln2 for every k. How are the corresponding JS-distance limits justified?
Q3. The invalid conditional-JS step in Lemma 3. In Appendix A.2, the authors define the joint mixture MBA=21(PBA+QBA). Its conditional distribution is MB∣A=a=PA(a)+QA(a)PA(a)PB∣A=a+QA(a)QB∣A=a, which is generally a marginal-dependent weighted mixture. However, the proof then identifies the corresponding conditional KL terms with JS(PB∣A=a∣QB∣A=a), whose reference mixture is MˉB∣A=a=21(PB∣A=a+QB∣A=a). These two mixtures are generally different when the marginal weights differ and the conditional distributions are distinct. Consequently, the identification in the proof of Lemma 3 is unjustified, and the claimed chain-rule inequality is not valid in general. Can you give the correct proof of Theorem 3?
Q4. The generalization guarantee for the data-dependent minimizer in Corollary 2. The proof applies a fixed-hypothesis Hoeffding bound to h=argminh∈HRλ(h) without uniform convergence or complexity assumptions. Consider X1∼U(0,1), X2∼U(1,2), and XT∼U(2,3), with Y≡0, G=g, and g(x)=x−⌊x⌋. All domains induce the same joint distribution PeY,g(X)=δ0⊗U(0,1), and Assumption 1 holds. Let F contain all measurable binary predictors, use squared loss (M=1), and choose a representation penalty satisfying N(g)=0 under exact marginal matching and invariance. For the finite set S of observed source representations, define fS(z)=0 if z∈S and fS(z)=1 otherwise. Then h=fS∘g is a valid regularized empirical-risk minimizer with R(h)=0 and N(g)=0, yet RT(h)=1 almost surely, while the optimal predictor se≡0 satisfies Re(se)=0 and N(g)=0. All joint JS discrepancy terms vanish, so Corollary 2 would imply 1≤2ln(1/δ)/(EN), which fails for sufficiently large EN. This construction satisfies the exact representation-matching and invariance conditions intended in Corollary 2. Since h is selected using the training data, a concentration bound for fixed hypotheses cannot be directly applied to it. Moreover, the counterexample shows that a valid empirical-risk minimizer can have zero training risk but unit target risk, even when the source and target joint distributions coincide. What additional assumptions, uniform convergence argument, or mathematically specified selection rule can justify Corollary 2?
Q5. The unsupported conclusion about DG versus ERM. The analysis provides bounds RT(hDG)−RT(sT)≤BDG and RT(hERM)−RT(sT)≤BERM, without establishing an ordering between BDG and BERM. However, incomparable upper bounds do not imply RT(hDG)≥RT(hERM), nor do they rule out improvements from DG algorithms. What mathematical result justifies the broader conclusion that DG methods have no theoretical advantage over ERM and that algorithmic improvements alone cannot lead to genuine breakthroughs?
From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.