acceptodds
Under review as a conference paper at ICLR 2027

Latent-Geometry-Aware Training for Domain Genearlization

Abstract

Learning-based decision-making models tend to underutilize information from less frequent scenarios, often resulting in poor domain generalization. Most existing domain generalization methods focus on learning domain-invariant representations, typically assuming access to domain information during training. In this work, we develop a fine-tuning strategy for generic classifiers that enhances domain generalization without requiring any domain information. The key idea is to first (i) jointly learn a feature extractor and a low-dimensional affine subspace that minimizes within-class latent dispersion while maximizing between-class latent dispersion, and then (ii) further fine-tune the model by minimizing within-class logit dispersion while maximizing the classification accuracy. Experiments on the ColoredMNIST, Waterbirds, and UrbanCars benchmarks show that the proposed fine-tuning approach achieves domain-generalization performance comparable to state-of-the-art methods that require domain information during training.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
domain generalization, representation learning, latent geometry, fine-tuning, robust classification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iVbHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a two-phase approach to domain generalization without explicit domain or group labels during training. The key assumption is that the training data include label-preserving matched sets that share a class but differ in a nuisance variation the model should ignore. The training is done in multiple phases (2), each finetuning the feature extractor and then a classifier.

Critical strengths
  1. The papaer addresses DG under worst-group performance without requiring explicit domain/group labels for every training example.
Critical weaknesses
  1. In the method the combination of subspace projection, representation learning, and logit-margin regularization is established. But the specific benefit of the proposed integration lacks stronger motivation.
  2. Although the method does not require explicit domain IDs for every training example, it uses matched examples or counterfactual variations to identify nuisance changes. It is still considered as a meaniful supervision. There needs to be a comparison to test performance on how much and what types of matched examples are useful.
  3. The results from Table. 1 shows that LISA beats the proposed method in two out of three benchmarks. In the first two the margin is substantial.
  4. There is mismatch in the result presented in table 1, and that explaiend in section 4.2.
  5. The generalizability of the proposed method is shown limitedly. Its performance across other popular DG benchmarks such as DomainNet has not been established.
  6. The main design choice of this method: tolerances controlling within-class logit dispersion and worst-case class margins. These hyperparameters are not ablated.
Questions for authors
  1. How much of the improvement comes specifically from the proposed latent-geometry learning? Do you have any ablation done in this regard?
  2. What is the total training and tuning cost, including the geometry updates and the construction of matched examples compared with standard ERM and other baselines?
  3. Can authors clarify about how much matched data would be needed and how robust is the method when there are imperfect or noisy matches?
1 comment
AuthorsOpenReview ↗

Response to Reviewer iVbH

During the rebuttal, (thanks to the reviewers’ comments) we realized that our formulation and experiments were inconsistent, making our “domain-label-free” claim overstated: although the algorithm takes neither domain label nor object-identity labels, during the experiments we utilized object information to construct the label-preserving pairs. This information leakage made our reported experimental comparison misleading, especially due to our choice of baselines that do not necessarily benefit from object-identify labels. During the rebuttal phase, we tried to address this gap: we evaluated label-preserving pair without any object-identity labels; matching our algorithm with the experimental results. Unfortunately, we observe a substantial amount of performance drop which we couldn’t resolve at the moment. We therefore agree with the reviewers’ shared opinion on this work that it should be rejected at least in this stage. We apologize again for our inconsistency; also for the delayed response. And most importantly, we appreciate the reviewers for their constructive comments/criticisms. In the following, we tried to provide point-by-point response to the reviewer’s comments, in case they are of the interest still.

Q1 Contribution of latent geometry learning

The ablation compares learning B on a fixed ERM representation with jointly updating B and the representation parameters in Eq. (7). All results below are measured right after projection, before adaptation. The gain persists when direct projection uses its natural rank, supporting joint representation learning beyond rank selection alone.

BenchmarkDirect matched rank (%)Direct natural rank (%)Reorganized (%)
ColoredMNIST35.6 ± 7.235.6 ± 7.251.4 ± 5.6
Waterbirds70.7 ± 0.070.7 ± 0.082.5 ± 0.4
UrbanCars34.9 ± 2.053.9 ± 3.087.2 ± 2.8

Q2 Training and data construction cost

The UrbanCars asset counts below include one label and one crop per car, with backgrounds and co-objects reused across rendered views. Beyond ERM, we learn the representation and adapt after projection; adaptation alone takes 69–148 minutes per UrbanCars sensitivity run.

SplitCars / labelsRendered viewsBackgroundsCo-objects
Training6,00024,0003,8011,997
Subspace selection2,0008,0002,5171,715
Calibration1,0004,0001,5831,272

Q3 Amount and type of useful matched data

On UrbanCars, reducing matched-set coverage to 10% lowers worst-group accuracy from 87.5% to 77.9%. Replacing views in 10% of the matched sets with views of different cars and different nuisance-source images lowers it to 49.3 ± 37.7%. Indeed, our algorithm is sensitive to such imperfect matches which we couldn’t resolve at the moment. Nonetheless, a good matched data for a new application might be obtained by cropping the object using some off-the-shelf object detection algorithm and then generate backgrounds using generative AI tools. Although this can be an interesting direction, it was beyond the scope of our revision; we will try to address the reviewer’s comments in our future work. Thanks again for raining important comments/criticisms.

Reviewer_PaNCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a two-phase training strategy for domain generalization by label-preserving matched sets instead of domain assignments. In Phase 1, the feature extractor is jointly trained with a low-dimensional subspace to concentrate specified variation in removable directions while preserving task-relevant structure. A reference-centered affine projection then removes sample-dependent variation along the identified subspace. In Phase 2, post-intervention adaptation aims to recover classification performance while constraining within-set logit variation and preserving worst-view classification margins. Experiments on ColoredMNIST, Waterbirds, and UrbanCars evaluate worst-group accuracy, robustness to multiple simultaneously varying factors.

Critical strengths
  1. The paper improves robustness to spurious correlations without requiring explicit domain or group labels.

  2. The method considers multiple simultaneously varying nuisance factors, extending its scope beyond single-factor settings.

  3. The experiments include component ablations and counterfactual sensitivity analyses, providing useful insights beyond aggregate classification accuracy.

Critical weaknesses
  1. Although the method does not require explicit domain labels, it relies on carefully constructed counterfactual pairs or sets that preserve the foreground object while changing nuisance factors. This is substantial additional supervision, and the paper does not establish whether its complex training procedure offers clear benefits over simpler ways of exploiting the same information, such as background randomization or pairwise consistency training.

  2. The baseline comparison is incomplete and largely focuses on older domain-generalization methods. More recent and directly relevant approaches, such as CFA and EvA, as well as the multi-shortcut method LLE, are missing, making the claimed competitiveness with state-of-the-art methods insufficiently supported.

  3. The evaluation is limited to three controlled benchmarks where nuisance factors can be explicitly manipulated to construct reliable matched examples. It remains unclear whether the method generalizes to more natural distribution shifts or remains effective when correspondences are imperfect.

  4. The performance gains are not consistent across datasets. According to Table 1, the method underperforms LISA in worst-group accuracy by 8.5 percentage points on ColoredMNIST and 4.0 points on Waterbirds; its main advantage appears on UrbanCars, where stronger and supervision-matched comparisons are still needed to establish whether the improvement justifies the additional supervision and methodological complexity.

Questions for authors
  1. How does the proposed method compare with ERM using background randomization and simple pairwise consistency training?

  2. How does the method compare with more recently relevant approaches?

  3. Is there evidence that the method remains effective with imperfect correspondences or domain shifts beyond explicitly manipulated nuisance factors?

1 comment
AuthorsOpenReview ↗

Response to Reviewer PaNC

During the rebuttal, (thanks to the reviewers’ comments) we realized that our formulation and experiments were inconsistent, making our “domain-label-free” claim overstated: although the algorithm takes neither domain label nor object-identity labels, during the experiments we utilized object information to construct the label-preserving pairs. This information leakage made our reported experimental comparison misleading, especially due to our choice of baselines that do not necessarily benefit from object-identify labels. During the rebuttal phase, we tried to address this gap: we evaluated label-preserving pair without any object-identity labels; matching our algorithm with the experimental results. Unfortunately, we observe a substantial amount of performance drop which we couldn’t resolve at the moment. We therefore agree with the reviewers’ shared opinion on this work that it should be rejected at least in this stage. We apologize again for our inconsistency; also for the delayed response. And most importantly, we appreciate the reviewers for their constructive comments/criticisms. In the following, we tried to provide point-by-point response to the reviewer’s comments, in case they are of the interest still.

Q1 Background randomization and pairwise consistency

We added background-only ERM with two views per car and four-view consistency training with a squared logit-difference penalty of weight 1, using 500 cars and three seeds. Consistency gives 90.1% worst-group accuracy versus 85.1% for ours, but 80.0% under alternative checkpoint selection. Batch grouping and training budgets differ: about 100 epochs for the controls versus 300 for four-view ERM.

MethodWorst group (%)Adjusted (%)Flip
Background-only ERM33.1 ± 2.075.5 ± 0.9.357 ± .026
Four-view ERM71.5 ± 1.787.1.134
ERM with consistency90.1 ± 0.994.0 ± 0.4.039 ± .005
Ours85.1 ± 4.491.5.043

Q2 More recent methods

Thank you for bringing the important references, CFA, EvA, and LLE, to our attention. We will consider these comparisons in our future work.

Q3 Imperfect correspondence and natural shifts

Replacing views in 10%, 25%, and 50% of UrbanCars matched sets with views of different cars and nuisance-source images gives worst-group accuracies of 49.3%, 58.4%, and 70.1%, versus 87.5% initially. Indeed, our algorithm is sensitive to such imperfect matches which we couldn’t resolve at the moment. Thanks again for raining important comments/criticisms.

Reviewer_Zk7pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a two-phase domain-generalization pipeline that avoids domain/group labels during training. Phase 1 jointly fine-tunes a feature extractor and an orthogonal low-dimensional projector so that specified nuisance variation is concentrated into a removable latent subspace while between-class structure is preserved. A reference-centered affine projection then removes this variation. Phase 2 adapts the classifier under decision-space constraints. Using label-preserving counterfactual matched sets, the method improves worst-group accuracy on ColoredMNIST, Waterbirds, and UrbanCars, particularly when multiple nuisance factors vary simultaneously.

Critical strengths
  1. The central idea that reorganizing latent geometry before erasure is well motivated. Ablations substantiate it and make the geometric-reorganization claim credible and distinguishes the method from standard representation-erasure approaches.

  2. Evaluation with interacting nuisance factors. Section 4.3 studies background and co-occurring objects on UrbanCars, including cases where both factors conflict with the label and diagnostics based on changing individual factors. This is a useful test of whether the intervention handles more than one nuisance.

Critical weaknesses
  1. The “no domain information” claim is overstated: the method still requires specified nuisance factors and label-preserving counterfactual pairs/orbits, whose acquisition is nontrivial and benchmark-specific. This weakens the practical advantage over domain-supervised DG.

  2. Key mechanisms are relegated to appendices; the main text gives little intuition for rank selection, commit gates, or constrained post-adaptation, making the method hard to assess. Moreover, the central objective and basis construction are inconsistent. For example, Eq. (6) appears inconsistent with the stated goal: Section 2.2.1 says harmful within-class variation should lie in span(B) and be removed by projection, but Eq. (6) constrains the squared projection of that variation onto span(B) to zero, placing it in the retained complement instead. Please clarify which formulation is used in the experiments and provide the corrected, consistent constraint.

Questions for authors

(1) How sensitive are results to the quality/coverage of counterfactual evidence, and how would one obtain it in a new application?

(2) What is the essential difference from LEACE/IPG beyond reorganizing before projection, ideally with a controlled comparison?

(3) Does the factorial-orbit construction scale beyond two binary nuisance factors?

(4) Inconsistent results and metrics: On page 7, ColoredMNIST worst-group/average accuracy is 61.7%/72.6% in Table 1 but 69.7%/71.6% in Section 4.2; Waterbirds average/worst-group is 94.8%/83.2% versus 91.7%/85.0%; UrbanCars adjusted/worst-group is 92.3%/86.9% versus 90.5%/82.7%. Please reconcile these.

(5) The paper provides no source-code availability statement. Given the complex commit gates, constrained post-intervention optimizer, and many frozen hyperparameters in Appendices C/E, this omission substantially hinders reproducibility and independent verification.

Suggestion: The main text would benefit from a method overview figure or concise pseudocode linking ERM, reorganization (rank selection/commit), affine projection, and constrained adaptation; currently these mechanisms are confined to Appendices C/E, making the pipeline hard to assess.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Zk7p

During the rebuttal, (thanks to the reviewers’ comments) we realized that our formulation and experiments were inconsistent, making our “domain-label-free” claim overstated: although the algorithm takes neither domain label nor object-identity labels, during the experiments we utilized object information to construct the label-preserving pairs. This information leakage made our reported experimental comparison misleading, especially due to our choice of baselines that do not necessarily benefit from object-identify labels. During the rebuttal phase, we tried to address this gap: we evaluated label-preserving pair without any object-identity labels; matching our algorithm with the experimental results. Unfortunately, we observe a substantial amount of performance drop which we couldn’t resolve at the moment. We therefore agree with the reviewers’ shared opinion on this work that it should be rejected at least in this stage. We apologize again for our inconsistency; also for the delayed response. And most importantly, we appreciate the reviewers for their constructive comments/criticisms. In the following, we tried to provide point-by-point response to the reviewer’s comments, in case they are of the interest still.

Correction to Eq. (6) and the projection

Thanks for catching this inconsistency. Indeed, Eq. (6) contains a typo – both terms should consider projections onto span(B)⊥span(B)^\perp.

Q1 Quantity and quality of matched evidence

In UrbanCars data set, we have observed that constructing label-preserving pair without using object information drops our performance more than 10%, demonstrating the importance of the quality of label-preserving pair as the reviewer correctly anticipated. For new applications, one may consider cropping the object based on some off-the-shelf object detection algorithm, and then generate different backgrounds using some generative AI tools.

Q2 Distinction from LEACE and IPG

LEACE aims at finding a good basis that removes undesirable information given a fixed representation. IPG aims at similar thing but relying on a more flexible gradient-based approach rather than a more structured (linear-algebraic) approach carried out by LEACE. We tried to design algorithm by taking the best-of-both-worlds – jointly learn both representation and basis using both linear projections as well as gradient-based updates.

MethodAvg (%)WG (%)Flip
LEACE-layer3 A192.8 ± 0.176.4 ± 0.4.083 ± .001
IPG88.8 ± 0.685.6 ± 1.0.066 ± .003
Ours94.8 ± 0.483.2 ± 0.4.027

Q3 More than two binary factors

In Appendix B, Eq. (20), extension beyond two binary factors are stated, though we have only tested with two factors in our experiments.

Q4 Inconsistent results in Table 1 and Section 4.2

This was a reporting error in Section 4.2; Table 1 contains the correct results. The correct worst-group / average accuracies are 61.7% / 72.6% for ColoredMNIST, 83.2% / 94.8% for Waterbirds, and 86.9% / 92.3% for UrbanCars. We will replace the outdated values and specify pooled averaging for ColoredMNIST and adjusted accuracy for UrbanCars.

###Q5 Reproducibility and code availability We will correct the main-text projector and basis description to match the intended formulation in Appendix C. An overview and pseudocode will specify ERM initialization, joint representation and basis learning, rank and commit checks, affine projection, and constrained adaptation. We will also add the missing code-availability statement.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Shared response

As the reviewers correctly identified, our formulation and experiments were inconsistent, making our “domain-label-free” claim overstated: although the algorithm takes neither domain label nor object-identity labels, during the experiments we utilized object information to construct the label-preserving pairs. This information leakage made our reported experimental comparison misleading, especially due to our choice of baselines that do not necessarily benefit from object-identify labels. During the rebuttal phase, we tried to address this gap: we evaluated label-preserving pair without any object-identity labels; matching our algorithm with the experimental results. Unfortunately, we observe a substantial amount of performance drop which we couldn’t resolve at the moment. We therefore agree with the reviewers’ shared opinion on this work that it should be rejected at least in this stage. We apologize again for our inconsistency; also for the delayed response. And most importantly, we appreciate the reviewers for their constructive comments/criticisms. In the following, we tried to provide point-by-point response to each reviewer, in case they are of the interest still.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.