acceptodds
Under review as a conference paper at ICLR 2027

Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations

Abstract

Traditional domain generalization (DG) methods commonly encourage domain-invariant representations to improve out-of-distribution (OOD) performance. However, we find that this does not transfer directly to finetuning CLIP, a widely used foundation model. Applying invariance regularization during finetuning gives little or no benefit on OOD data compared to standard finetuning. We hypothesize this is because invariance and the preservation of pretrained features can conflict with each other. Domain invariance discourages reliance on domain-specific features, but in foundation models these features were learned during pretraining and are part of what makes the model useful. Retaining domain-specific knowledge would require domain awareness, which standard invariance methods do not provide. To address this, we propose CLIP-DCA (**D**isentangling **C**lassification from enhanced domain **A**ware representations), which encourages invariance only at the classification head through disentanglement, while domain awareness is encouraged in the encoder using a separate domain head supervised by synthetic domain images and style descriptions generated by a multimodal LLM (MLLM). Rather than removing all domain information from the classifier, CLIP-DCA balances invariance with awareness, as we find that pushing the two heads further apart lowers accuracy. Since recent work shows that standard DG benchmarks can be contaminated by CLIP's web-scale pretraining, we evaluate on both established out-of-pretraining (OOP) splits and a 33-dataset benchmark indexed by a continuous multi-modal OOD score. CLIP-DCA shows improvements on both. Our results suggest that, for CLIP, traditional DG principles remain useful when invariance is balanced with domain awareness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Domain generalization, in the wild, CLIP, domain awareness, domain invariance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AgKnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CLIP-DCA for robust fine-tuning of CLIP. It adds a domain head and encourages separation from the classification head through soft orthogonality, using synthetic style images, MLLM descriptions, and hidden-state supervision. The method is evaluated on OOP splits and 33 target datasets, with additional unlearning experiments intended to assess dependence on pretrained domain knowledge.

Critical strengths
  1. Separating domain awareness from the classification pathway is a sensible design for preserving useful pretrained information. The auxiliary domain head is discarded at inference, retaining the standard CLIP prediction pipeline.

  2. The component ablations provide evidence beyond gains from additional data alone. In Table 4, keeping domain descriptions and MLLM hidden-state supervision, adding disentanglement improves average accuracy from 51.9%51.9\% to 55.1%55.1\%.

Critical weaknesses
  1. Generalization beyond CLIP pretraining is not established. The OOD score is relative to ImageNet, and unlearning is a proxy that also degrades general capabilities. Moreover, Tables 1 and 8 describe OOP by cosine similarity s<0.15s<0.15, whereas Teterwak et al. (ICLR 2025) discard that range before splitting at 0.210.21. The actual evaluation subsets and claims about reduced pretraining contamination need clarification.

  2. Superiority over strong baselines is not established under matched auxiliary supervision. Table 4 supports disentanglement, but Table 2 does not document matching GCC, synthetic data, and MLLM signals for the strong baselines. Table 5 matches GCC access and gives DANN the same images and descriptions, but does not establish matched hidden-state supervision.

Questions for authors
  1. What exact OOP filtering rules, backbone, and sample subsets were used in Tables 1 and 8? Please provide the split code or sample lists and clarify whether the threshold discrepancy is a reporting error. What evidence supports interpreting these evaluations as generalization beyond pretraining?

  2. Which auxiliary images, descriptions, and hidden-state signals does each baseline use? Do the strongest applicable robust fine-tuning baselines remain inferior when given the same auxiliary supervision as CLIP-DCA?

1 comment

Thank you for your careful and insightful review! We are encouraged that you found separating domain awareness from the classification pathway to be a sensible design and noted that our component ablations provide clear evidence beyond gains from additional data alone.

 

W1 / Q1. OOP splits and generalization beyond pretraining

  • Thank you for the careful check and the chance to clarify. We will correct our mislabeled captions.
  • We use the rule of Teterwak et al. In-Pretraining is above 0.21, Out-of-Pretraining is 0.15-0.21, and samples below 0.15 are discarded (as listed in Section 3.1). We use their released splits and alignment scores without changes (HF: PTeterwak/DomainBed_OOP), where we use leave-one-domain-out training.
  • Tables 1 and 8 use CLIP ViT-B/32 (Appendix C).
  • The sample counts are below:
DatasetIPOOPDiscarded
PACS8,6211,065305
VLCS4,7054,6921,332
OfficeHome14,4421,002144
TerraInc21,7722,50553
DomainNet424,752125,71345,541

 

  • On generalization beyond pretraining, we agree that alignment scores and unlearning are proxies. Unfortunately, limited resources make it difficult to do an entire round of pretraining. Instead, this is why we report two views, the OOP splits (Tables 1 and 8) and our OOD score against ImageNet on 33 datasets (Figures 6 and 7). CLIP-DCA improves over FLYP on both (OOP 55.3 vs 50.6, Table 8).

 

W2 / Q2. Matched auxiliary supervision

  • Thank you for this important question. We agree that the baselines should be compared under the same auxiliary supervision. The difficulty is that other methods have different assumptions for training. The table lists the data and signals each method uses in our paper (Tables 4 and 5).
MethodGCCSynthetic images + descriptionsHidden statesDisentanglementAvg. on 33 datasets
FLYP, ImageNet only (Table 5)----47.0
DANN, ImageNet only (Table 5)-✓--46.2
CLIP-DCA, ImageNet only (Table 5)-✓✓✓51.5
FLYP (Table 5)✓---51.8
DANN (Table 5)✓✓--50.5
CLIP-DCA without hidden states (Table 4)✓✓-✓53.0
CLIP-DCA without disentanglement (Table 4)✓✓✓-51.9
CLIP-DCA (Table 5)✓✓✓✓55.1

 

  • In Table 2, the baselines do not use GCC. They are finetuned on ImageNet with their default configurations (Appendix C). CLIP-DCA still performs better than other robust fine-tuning baselines (Table 5).
MethodImageNetImageNet-V2ImageNet-SketchImageNet-AImageNet-R
MMA (Table 2)71.560.036.518.556.0
WiSE-FT (Table 2)72.961.540.026.261.0
MIRO (Table 2)74.062.536.024.559.0
CLIP-OOD (Table 2)71.060.240.513.645.2
CLIP-DCA, ImageNet only (Table 5)75.264.040.525.563.1

 

Thank you again for your careful reading and helpful review!

Reviewer_bEY3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies the problem of typical domain generalization methods on CLIP, i.e., applying invariance regularization during finetuning gives little or no benefit on OOD evaluation. The paper proposes CLIP-DCA, a training pipeline that encourages invariance only at the classification head, and introduces the domain head for domain awareness, supervised by synthetic domain images and style descriptions generated by diffusion models and MLLMs. On OOP splits and benchmarking using 33 datasets, CLIP-DCA shows improvements in OOD performance.

Critical strengths
  1. The paper takes a different stance on the domain generalization problem by disentangling and only enforcing invariance on the classification head, which is well-motivated by the paper’s experiments.
  2. The experiments are comprehensive, covering 33 datasets and OOP subsets as well. CLIP-DCA shows superior performance in the evaluations.
  3. The overall method is reasonable and sound. The paper is well-written overall.
Critical weaknesses
  1. The scope of the paper seems to be limited, as the paper only examines and experiments on CLIP models (CLIP ViT-B/32 and ViT-B/16). While the paper’s focus is on CLIP models, constraining it to only the CLIP model family limits the impact and contribution of the paper. It would be beneficial and interesting to see and show whether the finding that “applying invariance regularization during finetuning gives little or no benefit on OOD data compared to standard finetuning” also generalizes to other VLMs. Experiments with other VLM families would help to expand the contribution and impact of the paper.
  2. The disentanglement loss currently only encourages orthogonality between the class-domain and class-style pairs. While the ablation supports each component design of the paper, the paper does not appear to have stronger evidence to establish and support the hypothesis that domain awareness is a prerequisite for effective invariance. Additionally, in Table 3, it appears that all the regularization terms are scaled together, which does not isolate the effectiveness of the disentanglement. The paper would benefit from a more isolated experiment and better evidence to show the effectiveness of disentanglement and the hypothesis that domain awareness is a prerequisite for effective invariance.
  3. I assume that the diffusion-generated samples, varying in style, are used to simulate domain shifts. This can be somewhat limited, as the domain shifts can involve other changes, i.e., background and image conditions. While the evaluation on 33 datasets covers many domain shift scenarios, the paper would benefit from a better justification for using varying style as the main source of synthetic domain variation.
  4. Some of the writing is hard to follow. The actual loss, i.e., disentanglement and contrastive loss, is only formally described in the appendix, with contrastive loss named as “classification loss” in Algorithm 2, making the paper difficult to follow at times. I think the paper would benefit from a rewrite for clearer definitions and more consistent terminology.
Questions for authors

Please see weaknesses.

2 comments

Thank you for your constructive review! We are encouraged that you found our approach well-motivated, reasonable, and sound, and our evaluation across the 33 datasets and OOP splits comprehensive.

 

W1 / Q1. Only CLIP models

  • Thank you for this valuable suggestion. We agree that other VLM families make our contribution broader, so we ran additional backbones, including SigLIP and EVA-CLIP. CLIP-DCA improves over zero-shot on DomainBed across these backbones.
BackboneZero-shot, OOPFLYP + GCC, OOPCLIP-DCA, OOPZero-shot, FullFLYP + GCC, FullCLIP-DCA, Full
SigLIP ViT-B/1661.2+4.9+6.770.3+3.2+6.3
EVA-CLIP ViT-B/1659.9+2.8+4.675.3+2.7+4.8

 

W2 / Q1. Isolating the disentanglement loss

  • Thank you for your careful reading. Please note that Table 4 isolates it by removing only the disentanglement loss. Table 6 (Appendix D) also scales the disentanglement terms separately (1e-4 to 1e-1) with the awareness weights fixed.
Method (Table 4)DescriptionsDisentanglementHidden statesAvg. on 33 datasets
FLYP---51.8
CLIP-DCA without disentanglement✓-✓51.9
CLIP-DCA✓✓✓55.1

 

  • Regarding "prerequisite", please note the breakdown in our paper:
    • A. Regular invariance (e.g. DANN) performs worse than regular finetuning. Our disentanglement loss without domain descriptions also performs worse than FLYP (50.7, Table 4).
    • B. We add domain awareness. However, without disentanglement, performance is similar to FLYP.
    • C. Balancing domain awareness with invariance works (55.1), though scaling the regularization further lowers performance (-1.4 at 100x, Table 3).
  • This is why we say that domain awareness is a prerequisite for effective invariance. However, we also agree that "prerequisite" can be interpreted too strongly. We will state our claim as a balance between domain awareness and invariance, as our abstract already does.
SettingAvg. on 33 datasets
FLYP (Table 4)51.8
A. DANN (invariance, Table 5)50.5
B. Descriptions (awareness), no disentanglement (Table 4)51.9
C. Descriptions (awareness) + disentanglement (invariance) = CLIP-DCA (Table 4)55.1

 

W3 / Q1. Style as the synthetic variation

  • Thank you for raising this point. We agree that a clearer justification for style is helpful, and we will add it to Section 2.2. We use style for two reasons:
    • Style defines the domains of common DG benchmarks. In PACS, OfficeHome and DomainNet, the domains are styles such as sketch, painting and clipart.
    • Style can be generated without class labels, so the synthetic domains are not biased toward specific classes (Sec. 2.2).
  • Please also note that our 33 test datasets include non-style shifts, such as camera traps, histopathology and satellite images (Table 7). We agree that style does not cover every shift, such as background and imaging conditions. Our goal was to create diverse domains, so that the model receives signals for both domain awareness and invariance. For example, please note that performance improves as we scale the number of synthetic domains (Table 11).

 

W4. Writing

  • Thank you for this concrete feedback. We agree that the losses should be consistently named. We will make the following changes:
    • We will give the contrastive and disentanglement losses as equations in Section 2.1. They are currently pseudocode in Algorithm 1 and Appendix B.
    • We will consistently say "contrastive loss" (and not "classification loss") in Section 2.1 and Algorithms 2 and 3, as in Algorithm 1.

Thank you again for your helpful review!

Thank you for the detailed response and additional experiments. I appreciate the authors’ efforts to address my concerns, particularly the additional evaluations on SigLIP and EVA-CLIP and the clarification of the disentanglement ablations. These additions strengthen the empirical evaluation and address several of my original concerns.

However, I still have some reservations. The additional backbones remain closely related to CLIP-style models, leaving broader generalizability unclear. Moreover, while the ablations support the effectiveness of disentanglement, the evidence for the specific proposed mechanism remains somewhat limited. Therefore, I decided to maintain my original rating.

Reviewer_ecdmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies a “domain generalization in-the-wild” setting where the goal is to fine-tune CLIP on a labeled source dataset with the goal of improved classification on diverse and unseen deployed target distributions. Empirical evidence in the work found that traditional invariance regularization methods does not appear to be beneficial over standard fine-tuning and hypothesized that invariance and the preservation of retrained features can conflict. Arguing that CLIP’s web-scale pre-training gives it rich features for the deployment domain. The work then claim that domain awareness in the encoder is a prerequisite for effective domain invariance at the classifier, which motivated their approach of disentanglement with the building of a domain head. The work also introduces a 4096 synthetic image dataset spanning 512 visual styles. These images are MLLM-generated. To avoid potential contamination from web-scale pre-training of CLIP, the authors evaluated on an established out-of-pretraining splits and a complementary 33-dataset evaluation. Ablations show that disentanglement without domain descriptions lowers performance, domain descriptions without disentanglement only provide small gains, and separating class and domain head lowers accuracy even more, thus supporting their hypothesis of including domain features. Their proposed method, CLIP-DCA, outperforms proprietary methods in 4 of 5 datasets and the OOD score shows strong negative correlation with the performance (accuracy) improvement, providing additional evidence to the central claims.

Critical strengths
  1. Very broad evaluation with a quantitive measure of distribution shift. The 33-dataset evaluation is diverse. In addition, the inclusion of a OOD score helps the reader’s appreciation of how different these datasets are.
  2. Explicit definition of the OOD score and per-sample penalty leaves nothing ambiguous.
  3. Systematic investigation using controlled synthetic domain.
  4. The ablations that were directly probes the proposed mechanism of CLIP-DCA. The experiments separately look at domain descriptions, disentanglement, domain diversity, corrupted or permuted descriptions, MLLM hidden states, and MLLM size. These help in distinguishing the proposed mechanism from gains vs simply additional data or larger capacity.
Critical weaknesses
  1. Evaluation setting differs substantially from the cited DG/DA settings. Conventional DG typically learns a classifier across labeled source domains for an unseen domain with the same label space; conventional closed-set DA additionally uses unlabeled target-domain data. The main evaluation here instead finetunes CLIP on ImageNet and transfers through text-based classification to datasets with different domains and class labels. This evaluates preservation of open-vocabulary transfer under finetuning, but does not isolate domain invariance from preservation of image–text alignment and novel-class knowledge. Therefore, the results provide limited support for broader conclusions about why established DG/DA methods fail. Instead, better evaluation could have been done without much changes to the experimental setup by simply keeping the class labels constant, instead of open-vocab.
  2. The adaptation of DA baselines may violate their intended assumptions. DANN, CDAN, DAN, CORAL, and MDD are evaluated using class-agnostic synthetic style images as the distributions to align, rather than samples from the deployment target distribution. For methods relying on shared class semantics or conditional alignment, this is a consequential change. Poor performance could reflect this mismatch rather than a conflict between invariance and pretrained knowledge. Matched conventional DG/DA experiments would help distinguish these explanations.
  3. The central mechanism is not established by the mathematical objective or probing results. Per-sample orthogonality between two learned outputs does not guarantee removal of domain information or statistical independence. Domain-label accuracy from the class head remains close to zero-shot CLIP: 80.1% versus 80.6%. The ablations demonstrate that the proposed components work together within this architecture, but do not establish that domain awareness is a prerequisite for invariance. Comparisons with explicit domain-subspace removal and downstream linear probing could clarify this claim.
  4. All reported accuracies and tables do not include random seeds or confidence intervals, the small performance differences are hard to assess. With the use of only once source dataset and two CLIP backbones, it is even harder to tell if the proposed method is broadly applicable.
Questions for authors

The main issue is that the evaluation setting is substantially different from the DG/DA setting. Perhaps the authors could clarify on why their comparison still makes sense and why the DA baseline assumptions from the cited works are not violated. This is the main issue that needs to be addressed. Secondly, the other major issue to address is the limited evaluation and thus the applicability of this method to anything other than CLIP.

2 comments

Thank you for your thoughtful review! We are encouraged that you found our 33-dataset evaluation and OOD score broad and explicit, and appreciated our systematic ablations probing the mechanism of CLIP-DCA.

 

W1 / Q1. Evaluation setting differs from classical DG/DA

  • Thank you for this important point. We agree that keeping the class labels constant is the most direct way to isolate the domain shift, and we want to make it clear that our paper does include this setting.
  • We finetune on ImageNet and test on ImageNet-V2, -Sketch, -A and -R (Table 2) on the same class set. CLIP-DCA performs best on most ImageNet datasets.
Method (Table 2)ImageNetImageNet-V2ImageNet-SketchImageNet-AImageNet-R
Zero-shot54.248.432.326.259.7
FLYP69.158.135.318.155.8
DANN69.058.033.118.353.7
WiSE-FT72.961.540.026.261.0
MIRO74.062.536.024.559.0
CLIP-DCA75.064.142.926.265.0

 

  • We also follow the classic DomainBed benchmark which tests on held-out domains (Tables 1 and 8). Classic methods, like DANN, use the real domain labels. CLIP-DCA still performs best on OOP in this setting. We will revise our paper to make this setting more visible by adding this to the start of Section 3.
Method (Table 8)FullIPOOP
Zero-shot65.270.647.8
FLYP69.774.950.6
DANN (real domain labels)69.874.550.6
CLIP-DCA73.077.955.3

 

  • Please also note why we additionally test on 33 datasets. We found that the domains of each DG benchmark cluster together (Figure 5), which suggests these benchmarks are not challenging enough for CLIP (Sec. 3.2). We therefore follow prior work on robust CLIP finetuning, which finetunes on ImageNet and tests on many datasets (Table 7).

 

W2 / Q1. DA baselines may violate their assumptions

  • Thank you for raising this point, as it gives us a chance to clarify the experiments. Tables 1 and 8 are a classical DG experiment. DANN uses real, labeled source domains with the same classes. CLIP-DCA still performs better in this setting (OOP 55.3 vs 50.6, Table 8).
  • For the ImageNet finetuning, there are no additional source domains (Sec. 2.2) as is the case for many real downstream tasks. This is why every method is given access to the same synthetic domains as CLIP-DCA (Sec. 4.2, Table 9).
Method (Table 9)Improvement over FLYP
DAN (MMD)+1.1
CDAN (Conditional adversarial)+0.5
CORAL (Covariance)+0.5
DANN (Adversarial)-0.3
MDD (Margin discrepancy)-3.6
CLIP-DCA+3.3

 

W3. Mechanism not established

  • Thank you for this careful reading and a chance to discuss further. We completely agree that orthogonality does not guarantee independence, and this is not our aim. As we note in Sec. 2.1 (line 220), both heads are single linear layers and the loss is a soft penalty. This is an intentional choice to bias the two heads apart without removing pretrained domain knowledge from the encoder. The class head staying close to zero-shot on domain probing (80.1 vs 80.6, Table 3) is consistent with this choice.
  • Regarding "prerequisite", please note the breakdown in our paper:
    • A. Regular invariance (e.g. DANN) performs worse than regular finetuning. Our disentanglement loss without domain descriptions also performs worse than FLYP (50.7, Table 4).
    • B. We add domain awareness through domain descriptions. However, without disentanglement, performance is similar to FLYP.
    • C. Balancing domain awareness with invariance works (55.1), though scaling the regularization further lowers performance (-1.4 at 100x, Table 3).
  • This is why we state that domain awareness is a prerequisite for effective invariance. However, we also agree that "prerequisite" can be interpreted too strongly. We will state our claim as a balance between domain awareness and invariance, as our abstract already does.
SettingAvg. on 33 datasets
FLYP (Table 4)51.8
A. DANN (invariance, Table 5)50.5
B. Descriptions (awareness), no disentanglement (Table 4)51.9
C. Descriptions (awareness) + disentanglement (invariance) = CLIP-DCA (Table 4)55.1

 

  • Following your valuable suggestion, we ran domain-subspace removal:
    • Our domain head is a single linear layer on the image feature, trained to match each synthetic image to its style description.
    • We take the SVD of the weight matrix of this layer, take the top 32 right singular vectors, and delete these 32 directions from the image feature before the class head (in training and at test).
    • These results suggest the removal of domain information (disentangled from classification) can indeed be helpful for learning domain generalizable features.
MethodOOP
FLYP (Table 8)50.6
FLYP + domain-subspace removal+1.4

 

W4 / Q2. One source dataset, two backbones

  • Thank you for this suggestion. We agree that more backbones make our results easier to assess.
  • We ran additional backbones, including SigLIP and EVA-CLIP. The table shows Full and OOP on DomainBed (leave-one-domain-out). Zero-shot is the accuracy, and the other columns are the gain over each backbone's zero-shot.
BackboneZero-shot, OOPFLYP + GCC, OOPCLIP-DCA, OOPZero-shot, FullFLYP + GCC, FullCLIP-DCA, Full
SigLIP ViT-B/1661.2+4.9+6.770.3+3.2+6.3
EVA-CLIP ViT-B/1659.9+2.8+4.675.3+2.7+4.8

 

Thank you again for your detailed review. We believe these additions make our paper stronger.

Comments on the paper

1 comment
Area_Chair_EntvOpenReview ↗

Discussion period ending soon

Dear Reviewers,

Thank you for your reviews of this submission. The authors have posted responses addressing the concerns raised.

With limited time remaining in the discussion period, please take a moment to review the authors' responses and post a brief follow-up indicating whether your concerns have been addressed or what remains outstanding. If your assessment changes as a result, please update your score accordingly.

Active engagement at this stage is essential for a fair and well-substantiated final decision. Thank you for your time and effort in reviewing this submission.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.