acceptodds
Under review as a conference paper at ICLR 2027

Chaos Is a LADDER: Domain Generalization Beyond Invariance via Reweighting

Abstract

Domain generalization (DG) aims to learn from multiple source domains and generalize to unseen target domains. Most DG methods pursue invariance: they seek a causal representation whose prediction rule is invariant across domains. This principle is effective when the causal mechanism is stable, but becomes restrictive when the domain itself modulates how causal content maps to the response. In this case, directly feeding domain style into the predictor can create misleading shortcuts, since style does not by itself cause the response. Yet the apparent chaos of multiple styles can become a ladder: style can locate the unseen target domain among source domains and guide which domain-dependent prediction rules should be trusted. We propose Latent Adaptive Domain Disentanglement and Environment Reweighting (LADDER), a fixed-model DG pipeline that learns causal/style representations, freezes the encoders, fits source-specific classifiers, and uses an unlabeled target-domain covariate set only at inference to compute weights over these fixed classifiers, with no target labels or model-state updates. We establish theoretical guarantees for source reweighting and validate LADDER on simulations, FMoW, and a location-grouped iWildCam protocol, with gains in overall and group-averaged accuracy.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
conditional invariance, reweighting, representation disentanglement, generalization, causal representation, style representation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3U65OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the domain generalization problem, where a model is trained on several source domains and expected to generalize to unseen target domains. A common approach is to learn domain-invariant representations. However, this paper argues that relying on a single invariant prediction rule is limited. The core assumption of the paper is that if two domains have similar domain-level style distributions, their prediction rules should also be similar. Therefore, instead of completely discarding domain-specific information, the model can use such information to identify which source-domain prediction rules are more relevant to the target domain.

With the above intuition, LADDER learns a content representation for label prediction and a style representation for domain characterization. It uses domain-adversarial training to suppress domain information in the content representation, while encouraging the style representation to preserve domain information. It then trains a separate classifier for each source domain and stores the corresponding style distribution as a domain fingerprint. At test time, after observing a set of unlabeled samples from the same target domain, LADDER estimates the target style distribution, measures its distance to the source-domain fingerprints, and reweights the fixed source-specific classifiers accordingly to make predictions.

Critical strengths
  1. Although leveraging domain-specific information to improve domain generalization is not entirely new, the paper provides a novel and convincing way of utilizing such information. In particular, LADDER uses the target domain’s style distribution to select and reweight source-specific prediction rules, rather than directly using style features for label prediction. I find this distribution-level routing perspective intuitive and compelling.

  2. The theoretical analysis supports the main idea of the paper. It gives a principled explanation of why style-based routing should work, which strengthens the motivation behind the proposed method.

Critical weaknesses
  1. The problem setting seems to be between domain generalization and domain adaptation. Although LADDER does not update model parameters at test time, it still needs a set of unlabeled target samples to estimate the target style distribution and change the weights of the source-specific classifiers. This is different from the standard DG setting, where the model usually makes predictions without using a batch of target-domain data to adjust its prediction rule. Therefore, the setting is closer to a target-aware setting, and also shares some similarity with unsupervised domain adaptation. The paper should explain more clearly why this setting is still considered domain generalization and how it differs from domain adaptation or test-time adaptation.

  2. The experimental comparison is not fully fair. LADDER uses a set of unlabeled target-domain samples at test time to estimate the target style distribution, while several standard DG baselines, such as ERM and IRM, make predictions independently for each target sample and do not use this extra target-domain information. The paper should include more protocol-matched comparisons or separate the standard DG baselines from target-aware methods.

  3. Since LADDER only reweights existing source-specific classifiers, its ability to extrapolate beyond the observed source domains may be limited. Although the paper includes an extrapolation experiment, the current evidence is insufficient to show that LADDER can generalize well to target domains that are substantially different from all source domains.

  4. The presentation of the main proposal is too brief, and the paper moves quickly from the motivation to the formal method, so the key intuition behind LADDER is not explained in enough depth. In contrast, the theoretical section is relatively long and detailed. I think the paper would be easier to follow if more space were given to explaining the core proposal and problem setting, while some of the theoretical details were moved to the appendix.

Questions for authors
  1. Could the authors better justify the considered setting and clarify its difference from unsupervised domain adaptation?

  2. When comparing with existing methods, could the authors clarify which experimental comparisons are protocol-matched and which are not?

  3. Could the authors provide more evidence on the extrapolation ability of LADDER?

Reviewer_Se4qOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the problem of domain generalization. The authors argue that when conditional prediction mechanisms differ across domains, forcing a model to learn a single invariant predictor may limit its generalization ability. To address this, they construct separate semantic and style encoders, each equipped with multiple source domain-specific classification heads. During the test phase, the style branch is used to calculate the similarity between target and each source domain, which are then used to compute a weighted combination of domain-specific classification. Experimental results demonstrate that the proposed method can improve domain generalization performance.

Critical strengths
  1. Using multiple classification heads is a lightweight ensemble learning approach that is widely recognized as effective.

  2. The paper contains extensive mathematical derivations.

  3. Experimental results validate the effectiveness of the proposed method.

Critical weaknesses
  1. The novelty of LADDER over existing domain generalization methods is not clear. The idea of adapting predictions on the target domain based on statistics collected from the source domains has already been explored in numerous studies. Although the authors distinguish LADDER from some related methods in the Introduction, they do not clarify what advantages LADDER offers over them. Moreover, directly adapting predictions using the means and variances of batch normalization layers is a more common and lightweight alternative, which has been widely adopted in prior work. The authors should more clearly articulate the substantive contributions of LADDER beyond existing approaches and provide a compelling justification for why the proposed method is necessary.

  2. The lack of logical coherence makes the paper difficult to read. Taking the Abstract as an example, my understanding is that Lines 14–15 first point out the limitations of only relying on domain-invariant features when domain-specific features are beneficial. However, the subsequent statement that training with domain-specific styles performs poorly appears contradictory and leaves the reader confused. The authors should establish a clearer logical flow and explain the relationship between their claims.

  3. Key claims that are not yet well established require sufficient evidence. For example, Lines 53–58 seem to suggest that the style of an individual target sample is detrimental, whereas the style of the entire target domain is beneficial. However, this claim cannot be substantiated by argumentation alone and requires validation. Even if the underlying premise holds, how the number of target-domain samples affects model performance? Specifically, is there a threshold below which they become detrimental? Furthermore, does the claimed benefit persist when the target domain exhibits substantial style heterogeneity? All these issues were triggered by an abrupt claim. The authors should treat their claim more carefully and use a more rigorous analysis to substantiate them.

Questions for authors
  1. Please see weaknesses.
  2. Is there any verification of the performance on the DomainBed?
Reviewer_GmYsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work explores domain generalization where the relationship between causal content and labels varies across domains, and that enforcing a single invariant representation and predictor discards useful information, and domain-level information can indicate prediction rules. Controlled simulations provides supporting evidence for this claim. The proposed methods, LADDER, learns separate content and style representation using both task and domain prediction and adversarial suppression of domain information in content and a penalty term that discourages overlap between representation. Encoders are then frozen, and one classifier per source domain are trained on the extracted features. Each source’s style distribution are also stored. For inference, an unlabeled target domain’s style distribution is compared with the stored distributions using Sinkhorn distances, and the nearest sources are used, with their classifier logins averaged. Across 100 synthetic repetitions, LADDER achieves 72.88% interpolation accuracy and 86.23% extrapolation accuracy, a significant versus 67.64% and 82.79% for ARM-CML.

Critical strengths
  1. The paper explores an interesting direction in domain generalization by allowing prediction rules to vary across domains. This formulation permits the conditional rule P(Y∣Zc,e)P(Y\mid Z_c,e) to depend on the domain, of which the invariant-rule setting is a special case. The FMoW diagnostic provides empirical support for this idea, suggesting that style similarity contains transferability information.
  2. The paper includes well-designed routing ablations, such as uniform weighting, nearest-source selection, sample gating, and group gating. These ablations reuse the same frozen encoders and source classifiers, allowing differences to be more directly attributed to the routing mechanism. Additionally, paired-seed comparison are also included and the are transparent with the fact that some gates have intervals containing zero. Good reporting honesty here.
  3. Paper provides extensive theoretical analysis which help explain the conditions under which source reweighting can improve target-domain predictions. Under smoothness and generalized-linear-model regularization assumptions, Theorems 2-3 bound target parameter and excess risk with interpretable contributions from source-classifier estimation, style-distribution estimation and source coverage. Several remarks and corollarys also clearly establish the limitations and conditions of the claims made.
Critical weaknesses

The theoretical guarantees rely on oracle representations and assumptions not explicitly enforced during training.

The theoretical comparisons are limited to simple predictors. i.e. Corollary 1 compares LADDER with an additive pooled predictor that cannot capture interactions between content and domain information. Corollary 2 compares it with a fixed predictor that differs from the target prediction rule. It is unclear whether LADDER would retain its advantages over more expressive pooled or domain-adaptive models.

The experimental evaluation includes limited comparisons with recent DG methods. Additional comparisons on FMoW-WILDS and iWildCam-WILDS would better establish LADDER's competitiveness.

Although Theorem 2 accounts for finite-sample estimation error, the empirical robustness of LADDER to different target-set sizes remains unclear. Additional experiments with smaller unlabeled target sets would help evaluate its effectiveness under limited target-data availability.

The visualization provides qualitative evidence of causal/style separation but there is a lack of quantitative analysis of the learned representations. In particular, it remains unclear whether the style representation contains class information that could influence source reweighting.

The real-world evaluation is limited to FMoW-WILDS and iWildCam-WILDS. Additional experiments on datasets with different types of distribution shifts, such as PACS, OfficeHome, or DomainNet, would help establish whether LADDER generalizes beyond the evaluated environments.

Questions for authors
  1. Theory and training connections. Corollary 3 extends the analysis to imperfect representations, could the authors expand and clarify on how the training objective supports the theoretical assumptions?
  2. Generalizability over other distribution shifts. Could the authors also comment or evaluate LADDER on other DA or DG datasets like OfficeHome, PACS, or DomainNet or discuss why the method may not be applicable (if so). This can help gain insights on cases where we do not have that many domains to generalize to.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response: Summary of Revisions and Additional Results

We thank the three reviewers for their careful reading and constructive comments. In response to the reviewers' suggestions, we have revised the manuscript and added experiments and theoretical analysis. The main updates are summarized below.

  1. Additional real-world benchmarks. We added experiments on OfficeHome and DomainNet using the DomainBed leave-one-domain-out protocol. LADDER performs comparably to CORAL, the best-performing baseline on OfficeHome, and outperforms EQRM, the best-performing standard DG baseline in our DomainNet comparison. The results are reported in Section 4.3, Table 3, with experimental details in Appendix A.3.

  2. Additional DG baseline. We added comparisons with Arith (CVPR 2025) on FMoW-WILDS, iWildCam-WILDS, and in simulation. The results are included in Sections 4.1–4.2 and Appendix A.2.1.

  3. Target-summary size ablation on real data. We added a target-summary size ablation on FMoW-WILDS, evaluating LADDER with different numbers of unlabeled target samples. The results show that LADDER maintains similar accuracy even with a single target sample, while larger target summaries mainly improve stability. See Appendix A.2.5, Figure 15.

  4. Extended extrapolation experiments. We expanded the simulation to evaluate target domains beyond both boundaries of the source range. LADDER maintains its accuracy advantage over the compared non-oracle methods across the tested target coordinates, while its target-parameter estimation error increases with extrapolation distance. See Appendix A.2.1, Figure 7.

  5. Additional theoretical analysis. We introduced Theorem 4 to characterize when separate target-group style summaries improve prediction under heterogeneous target domains. We also clarified the assumptions underlying the theoretical guarantees and the role of the training objective in learning causal and style representations. These revisions appear in Sections 2.2, 3, and 3.3.

  6. Clarifications to the methodology and experimental protocols. We revised the Abstract and Introduction to clarify the motivation for LADDER, the distinction between sample-level style features and domain-level style distributions, and the differences from domain adaptation and test-time adaptation. We expanded the training and inference overview in Section 2, clarified target-data access conditions and protocol-matched comparisons in Section 4, and highlighted the existing representation diagnostics in Appendix A.2.2. We also moved the full statements of Corollaries 1 and 2 to Appendix B.3 to improve readability.

We appreciate the reviewers' feedback and welcome further comments or questions regarding the revised manuscript.

Area_Chair_SpiEOpenReview ↗

Regarding figure 1

Dear authors,

Thanks for your submissions. I quickly read your paper and may have one clarification question regarding the problem setup.

So basically, figure 1 reveals that

Old causal structure in DG was e -> (some latent variable) X_e -> Y_e and Z_c ->X_e and Z_c->Y_e

I feel confused about the new assumption, indeed, e->Y_e it's not unreasonable. However, if e is the cause of the label, the generalization can be technically impossible in the worst case. For example, if e have a strong control over Y_e and the new environment is unknown it's technically impossible to generalize. So how the assumptions/setting solve this?

I only read the intro and figure 1 and may omit important details so it's just my confusion rather than any concern....

AuthorsOpenReview ↗

Response to Area Chair SpiE

Thank you for the question. In Figure 1, e→Yee \to Y_e means that the prediction rule ηe(Y∣Zc)\eta_e(Y \mid Z_c) varies across domains. You are right that if the domain could act on the label arbitrarily, generalization would be impossible in the worst case. LADDER therefore restricts how the domain acts on the label.

Recent DG work also allows domain-dependent prediction rules and avoids the worst case through structural restrictions [1–3]. These works mostly require domain information to be supplied as known variables. Figure 1 states this dependence in one structural causal model, with the invariant-rule setting as a special case.

LADDER's restriction is Eq. (1): domains with similar style distributions have similar prediction rules. The effect of ee on YeY_e then varies smoothly with the style distribution, which is observable from unlabeled target samples. LADDER combines the prediction rules of the source domains closest to the target in style, so the domain may strongly affect the label as long as the target resembles some sources. For a target far from all sources, our bound grows with this distance (Section 3, Remark 1), consistent with the worst case you describe. Eq. (1) can be checked on source domains alone: on FMoW, style-distribution distance correlates with the transfer loss between source classifiers (Spearman ρ=0.53\rho = 0.53, Appendix Figure 12).

The main contribution of LADDER is that domain information about the target is learned from its unlabeled covariates rather than supplied, at small cost. Beyond the shared encoders, LADDER stores one classifier and a cached sample of style representations per source domain, 6.83 MiB on FMoW and 54.75 MiB on iWildCam in total (Appendix Table 14).

References

[1] Zhu, Y., Deng, N., Shi, N., Gangrade, A., and Scott, C. "Domain Generalization Under Posterior Drift." arXiv:2510.04441v2 (2026).

[2] Li, S., and Zhang, L. "Multi-Dimensional Domain Generalization with Low-Rank Structures." Journal of the American Statistical Association (2025).

[3] Londschien, M., Burger, M., Rätsch, G., and Bühlmann, P. "Domain Generalization and Adaptation in Intensive Care with Anchor Regression." arXiv:2507.21783v2 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.