acceptodds
Under review as a conference paper at ICLR 2027

Towards Unified Source-Free Domain Adaptation via Causality-Inspired Discovery of Latent Invariant Factors

Abstract

In the pursuit of transferring a source model to a target domain without access to the source training data, Source-Free Domain Adaptation (SFDA) has been extensively explored across various scenarios, including Closed-set, Open-set, Partial-set, Open-partial-set, and Generalized settings. Existing methods, focusing on specific scenarios, not only address a limited subset of challenges but also necessitate prior knowledge of the target domain, significantly limiting their practical utility and deployability. In light of these considerations, we introduce a more practical yet challenging problem, termed unified SFDA, which comprehensively incorporates all specific scenarios in a unified manner. In this paper, we propose a novel approach causality-inspired discovery of latent invariant factors for unified SFDA (CausalDA). In contrast to previous alternatives that emphasize learning the statistical description of reality, we formulate CausalDA from a causality perspective. The objective is to uncover potential causality-induced connections between latent invariant variables and model decisions, enhancing the reliability and robustness of the learned model against domain shifts. To integrate extensive world knowledge, we leverage a pre-trained vision-language model such as CLIP. This aids in the formation and discovery of latent invariant factors in the absence of supervision in the variation of distribution and semantics, coupled with a newly designed self-supervised invariant information maximization with theoretical guarantees. Extensive experiments demonstrate that CausalDA can achieve new state-of-the-art results in distinct SFDA settings, as well as source-free out-of-distribution generalization.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Unified Source-Free Domain Adaptation, Causal Inference, Vision-Language Models, Information Bottleneck

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ykQBOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the problem of unified adaptation in Source-Free Domain Adaptation (SFDA). While existing SFDA methods are typically designed for specific source-target category relationships—such as closed-set, open-set, partial-set, or generalized scenarios—this paper proposes ''Unified SFDA''. The goal is to handle diverse SFDA scenarios using a single unified approach, even in the absence of prior information about the target domain and without knowing the source-target category relationship beforehand. To this end, the authors propose CausalDA, which adopts a causality-inspired perspective to decompose latent invariant factors into two components: external and internal. External invariant factors are identified by leveraging a frozen vision-language model through prompt learning and objectives based on mutual information. Internal invariant factors are utilized to update the target model via unsupervised information maximization and pseudo-label guidance derived from the external factors. These two stages are optimized alternately. Experiments cover closed-set, generalized, open-set, and partial-set SFDA, as well as source-free OOD generalization. The results demonstrate that the proposed method achieves strong and relatively stable performance across these diverse settings.

Critical strengths
  1. The problem addressed in the paper holds clear practical significance. Existing SFDA methods typically rely on predefined source-target category relationships and employ different approaches for different settings. This paper aims to reduce such setting-specific dependency, enabling a single method to maintain stable performance across various SFDA scenarios. I consider this research direction valuable, particularly for real-world deployment where the category composition of the target domain may not be accurately known in advance.

  2. The method design is comprehensive, and the experimental results are highly convincing. The proposed ''external/internal invariant factor decomposition'' is realized through a concrete two-stage optimization process rather than remaining merely at a conceptual level. The experiments go beyond a single closed-set benchmark, systematically covering various SFDA settings and OOD generalization scenarios; the results demonstrate that CausalDA is highly competitive in these contexts. Furthermore, the paper presents thorough analyses, including ablation studies, pseudo-label analysis, invariance analysis, and prompt sensitivity tests. Overall, these results incline me to accept the paper.

Critical weaknesses

The core claim of ''Unified SFDA'' has not yet been established with sufficient clarity across the problem formulation, method description, and evaluation protocol. The paper characterizes ''Unified SFDA'' as adaptation performed without knowledge of the target domain or even the source-target category relationship. While this objective is compelling, it remains unclear to me just how ''unified'' the proposed method truly is in terms of practical deployment.

The algorithm incorporates an entropy- or K-Means-based rejection mechanism for unknown target classes. However, the paper fails to clarify whether this mechanism is consistently active across all settings or if prior knowledge regarding the presence of unknown classes is required. If the method relies on setting-specific activations, decision rules, or evaluation-specific processing, then the claim of operating ''without prior knowledge of the target setting or category relationship'' may be overstated.

Therefore, I believe the paper needs to make a clearer distinction between:

  • a single framework performing well across multiple known SFDA settings, and
  • a single deployment procedure operating without knowledge of the specific SFDA setting it faces.

While the paper provides ample evidence for the former, the definition and validation regarding the latter remain insufficiently direct. Given that ''Unified SFDA'' is a primary contribution of the work, this distinction is crucial to my scores.

Questions for authors

The manuscript also contains a clear quality-control issue in the Ethics Statement, which discusses ''malicious client detection in federated learning,'' a topic unrelated to this paper. It is maybe a simle typo copied from another manuscript. It was not corrected before submission, which raises some concern about the overall level of proofreading and manuscript preparation.

1 comment

Q1. I believe the paper needs to make a clearer distinction between: a single framework performing well across multiple known SFDA settings, and a single deployment procedure operating without knowledge of the specific SFDA setting it faces.

R: Thank you for this helpful suggestion. In this paper, “unified” refers to the proposed method’s ability to perform well across multiple SFDA settings, consistent with the reviewer’s first interpretation. We have clarified this meaning in the revised manuscript.

Additionally, SFDA allows access to unlabeled target data. Thus, the problem setting is already specified by the training data used in each experiment. Our statement “without prior knowledge of the target setting or category relationship” was intended to indicate that our method does not require an explicit setting label or identifier. We have removed this statement to avoid ambiguity regarding the meaning of “unified.”

∗∗Q2.∗∗Themanuscriptalsocontainsaclearquality−controlissueintheEthicsStatement,whichdiscusses′′maliciousclientdetectioninfederatedlearning,′′atopicunrelatedtothispaper.Itismaybeasimletypocopiedfromanothermanuscript.Itwasnotcorrectedbeforesubmission,whichraisessomeconcernabouttheoveralllevelofproofreadingandmanuscriptpreparation.∗∗R∗∗:Thankyouverymuchforpointingoutthistypoandapologizeforthisoversight.Wehavefixedit. **Q2.** The manuscript also contains a clear quality-control issue in the Ethics Statement, which discusses ''malicious client detection in federated learning,'' a topic unrelated to this paper. It is maybe a simle typo copied from another manuscript. It was not corrected before submission, which raises some concern about the overall level of proofreading and manuscript preparation. **R**: Thank you very much for pointing out this typo and apologize for this oversight. We have fixed it.
Reviewer_svuQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CausalDA, a source-free domain adaptation framework intended to accommodate different source–target category relationships while preserving source-domain knowledge. The method alternates between prompt optimization in a frozen vision-language model and adaptation of a source-initialized target model. The first stage combines a variational logit-alignment term with a prediction-based mutual-information objective; the second uses entropy/diversity regularization and supervision from the adapted vision-language model. These stages are interpreted as discovering external and internal invariant factors that approximate causal factors. Experiments cover several adaptation settings and ImageNet distribution shifts. The reported results suggest a promising balance between adaptation and source-domain knowledge retention. Further clarification of the theoretical formulation and the unified evaluation would strengthen the paper.

Critical strengths
  1. A practically relevant evaluation objective. The paper considers adaptation quality together with source retention and changes in category relationships. Appendix D reports a shared loss-hyperparameter configuration across settings, and Table 7 includes comparisons with DIFO and ProDe, which also exploit vision-language knowledge. CausalDA improves on ProDe in the reported open-set, partial-set, and SF-OODG results while remaining competitive in closed-set adaptation. Evaluating this broader trade-off is valuable even when a method is not uniformly best in every setting.
  2. Useful evidence for complementary adaptation components. Table 12 includes internal-only, external-only, zero-shot-CLIP-guided, and phase-1-only variants. The full method improves over the zero-shot-CLIP-guided variant from 82.0 to 84.3 on closed-set Office-Home and from 44.2 to 48.4 on ImageNet-Sketch. These comparisons support the utility of adapting the external guidance jointly with the target model. Additional studies of prompt initialization, alternative losses, and training resources provide useful insights into the effectiveness and practical characteristics of the proposed framework.
Critical weaknesses

1.Clarification of the optimization formulation. The phase-1 objective would benefit from further clarification. In particular, Eq. (6) resembles a Gaussian negative log-likelihood, whereas Eq. (9) maximizes this term with a positive coefficient. The treatment of the variational bound in Eq. (5) and the mutual-information estimation in Eq. (7) also requires clarification. Please verify the mathematical expressions and specify the exact objective used in implementation.

2.Interpretation of causal and invariant factors. The proposed causal perspective is interesting, but the relationship between the learned invariant factors and the underlying causal factors could be explained more clearly. In particular, the assumptions concerning logit-space interventions and the decomposition of internal and external factors deserve further justification. The inconsistency between Eq. (4) and Eq. (14) should also be corrected. A clearer distinction between causal motivation and empirically validated invariance would strengthen the theoretical presentation.

3.Completeness of the unified evaluation. The unified SFDA formulation explicitly includes the Open-partial-set setting in Table 1, but the experimental evaluation does not report dedicated results for this setting. Including such results would provide more comprehensive evidence for the proposed unified framework. It would also be helpful to clarify how the unknown-class rejection mechanism operates across different settings without prior knowledge of the target category relationships.

4.Clarification of the SF-OODG evaluation protocol. The SF-OODG experiments would benefit from a clearer description of the data-access protocol. Since the framework updates the target model using unlabeled data, please clarify which ImageNet-variant samples are used for adaptation and which are reserved for evaluation. Additional details regarding category information and the implementation of CLIP-based comparison methods would help establish a consistent experimental setting.

5.Consistency of experimental results and reporting. Several numerical results and descriptions should be checked for consistency. For example, Tables 3 and 9 report different CLIP accuracies on VisDA, while the stated improvement margin of at least 10.5 percentage points in the SF-OODG comparison does not appear to match Table 6. The Ethics Statement also appears to describe an unrelated federated learning task. Correcting these issues would improve the accuracy and clarity of the presentation.

Questions for authors

Q1. Could the authors clarify the exact phase-1 optimization objective used in the experiments, particularly the sign conventions in Eqs. (6) and (9), and provide more details on the mutual-information estimation?

Q2. Could the authors further explain how the learned invariant factors relate to causal factors, and clarify the assumptions underlying the internal-external factor decomposition?

Q3. Could the authors provide results for the Open-partial-set setting and clarify how the proposed framework handles different category relationships without prior knowledge of the target setting?

Q4. Could the authors clarify the adaptation and evaluation protocols for SF-OODG, particularly the use of unlabeled target samples and the information available to the compared methods?

Q5. Could the authors verify the inconsistencies in the reported CLIP results and improvement margins, and clarify the reliability of the relatively small performance gains in some settings?

5 comments

Dear Reviewer svuQ,

As our response slightly exceeds the word limit, we are submitting it in two parts: Part-1 addresses Q1–Q3, and Part-2 addresses Q4–Q5. The Part-1 is provided below:

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−∗∗Q1∗∗.Couldtheauthorsclarifytheexactphase−1optimizationobjectiveusedintheexperiments,particularlythesignconventionsinEqs.(6)and(9),andprovidemoredetailsonthemutual−informationestimation?∗∗R∗∗:Appreciateit.Aftercheckingtheequationsforphase−1,weidentifiedmissingnegativesignsinEqs.(6)and(7)andbeforetheKLterminEq.(5).Accordingly,theobjectiveinEq.(9)isupdatedinaminimizationmannerwithanegativesign.Regardingmutual−informationestimation,ourmethodinvolvesEqs.(5)and(7),whosedetailedderivationscanbefoundin[1]and[2],bothofwhichhavebeencitedintheprevioussubmission.[1]Theimalgorithm:avariationalapproachtoinformationmaximization.NeurIPS2004.[2]Invariantinformationclusteringforunsupervisedimageclassificationandsegmentation.ICCV2019. ------------------------------------------------------------ **Q1**. Could the authors clarify the exact phase-1 optimization objective used in the experiments, particularly the sign conventions in Eqs. (6) and (9), and provide more details on the mutual-information estimation? **R**: Appreciate it. After checking the equations for phase-1, we identified missing negative signs in Eqs. (6) and (7) and before the KL term in Eq. (5). Accordingly, the objective in Eq. (9) is updated in a minimization manner with a negative sign. Regarding mutual-information estimation, our method involves Eqs. (5) and (7), whose detailed derivations can be found in [1] and [2], both of which have been cited in the previous submission. [1] The im algorithm: a variational approach to information maximization. NeurIPS 2004. [2] Invariant information clustering for unsupervised image classification and segmentation. ICCV 2019.

Q2. Could the authors further explain how the learned invariant factors relate to causal factors, and clarify the assumptions underlying the internal-external factor decomposition?

R: We appreciate this comment. We relate the learned invariant factors to causal factors from two perspectives:

  • First, we construct an SFDA-specific structural causal model in the logit space, interpreted as a post-intervention space, where invariant factors are intended to capture causal semantics (Lines 158–164).
  • Second, the invariance analysis in Sec. 4.3 provides indirect support: CausalDA remains relatively stable as noise increases (Fig. 5), suggesting invariance consistent with causal explanations.

In addition, our internal–external factor decomposition is a modeling assumption rather than a theoretically guaranteed causal factorization. It is motivated by the difficulty of identifying all relevant causal relationships from source and target domains given limited data and domain-specific knowledge (Lines 180–182).

More importantly, we have already acknowledged that the learned invariant factors are not mathematically guaranteed to correspond to causal factors. Accordingly, we characterize our method as causality-inspired, and this limitation is explicitly discussed in Appendix H.

In addition, the inconsistency between Eq.(4) and (14) has been fixed by changing vdv_d to vev_e.

∗∗Q3∗∗.CouldtheauthorsprovideresultsfortheOpen−partial−setsettingandclarifyhowtheproposedframeworkhandlesdifferentcategoryrelationshipswithoutpriorknowledgeofthetargetsetting?∗∗R∗∗:Greatsuggestion!Weevaluatetheproposedmethodontheopen−partial−setsetting,followingtheexperimentalconfigurationin[1].Theresultspresentedinthetablebelowdemonstratetheadvantageofourmethod.Wehaveaddedtheresultstotherevisedversion.∗∗Table7.∗∗Open−partial−setresults(∣Method∣Venue∣Ar→Cl∣Ar→Pr∣Ar→Re∣Cl→Ar∣Cl→Pr∣Cl→Re∣Pr→Ar∣Pr→Cl∣Pr→Re∣Re→Ar∣Re→Cl∣Re→Pr∣Avg.∣∣:−−∣:−−:∣:−−:∣:−−:∣:−−:∣:−−:∣:−−:∣:−−:∣:−−:∣:−−:∣:−−:∣:−−:∣:−−:∣:−−:∣:−−:∣∣USFDA[1]∣CVPR20∣63.4∣83.3∣89.4∣71.0∣72.3∣86.1∣78.5∣60.2∣87.4∣81.6∣63.2∣88.2∣77.0∣∣EGRS[2]∣TMM26∣74.5∣80.8∣90.2∣71.6∣83.1∣90.1∣77.7∣54.8∣89.8∣81.0∣63.6∣86.5∣78.6∣∣PRIME[3]∣ESWA26∣71.2∣85.6∣91.6∣81.5∣83.0∣85.2∣81.1∣72.9∣∗∗92.2∗∗∣82.2∣74.2∣88.5∣82.4∣∣∗∗CausalDA−C−RN∗∗∣–∣72.9∣88.4∣∗∗91.9∗∗∣83.8∣88.0∣91.6∣85.4∣74.8∣91.9∣84.9∣75.0∣87.7∣84.7∣∣∗∗CausalDA−C−B32∗∗∣–∣∗∗80.6∗∗∣∗∗89.7∗∗∣91.4∣∗∗84.8∗∗∣∗∗89.6∗∗∣∗∗91.4∗∗∣∗∗86.5∗∗∣∗∗82.0∗∗∣91.5∣∗∗86.7∗∗∣∗∗82.3∗∗∣∗∗88.6∗∗∣∗∗87.1∗∗∣[1]Universalsource−freedomainadaptation.InCVPR2020.[2]Exploringgenericknowledgeandreactivatingsourcemodelforsource−freeuniversaldomainadaptation.IEEETMM,2026.[3]Pre−trainedinitializationandmemory−enhancedcorrectionforsource−freeuniversaldomainadaptation,ESWA,2026. **Q3**. Could the authors provide results for the Open-partial-set setting and clarify how the proposed framework handles different category relationships without prior knowledge of the target setting? **R**: Great suggestion! We evaluate the proposed method on the open-partial-set setting, following the experimental configuration in [1]. The results presented in the table below demonstrate the advantage of our method. We have added the results to the revised version. **Table 7.** Open-partial-set results (%) on Office-Home. | Method | Venue | Ar→Cl | Ar→Pr | Ar→Re | Cl→Ar | Cl→Pr | Cl→Re | Pr→Ar | Pr→Cl | Pr→Re | Re→Ar | Re→Cl | Re→Pr | Avg. | |:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| | USFDA [1]| CVPR20 | 63.4 | 83.3 | 89.4 | 71.0 | 72.3 | 86.1 | 78.5 | 60.2 | 87.4 | 81.6 | 63.2 | 88.2 | 77.0 | | EGRS [2]| TMM26 | 74.5 | 80.8 | 90.2 | 71.6 | 83.1 | 90.1 | 77.7 | 54.8 | 89.8 | 81.0 | 63.6 | 86.5 | 78.6 | | PRIME [3]| ESWA26 | 71.2 | 85.6 | 91.6 | 81.5 | 83.0 | 85.2 | 81.1 | 72.9 | **92.2** | 82.2 | 74.2 | 88.5 | 82.4 | | **CausalDA-C-RN** | – | 72.9 | 88.4 | **91.9** | 83.8 | 88.0 | 91.6 | 85.4 | 74.8 | 91.9 | 84.9 | 75.0 | 87.7 | 84.7 | | **CausalDA-C-B32** | – | **80.6** | **89.7** | 91.4 | **84.8** | **89.6** | **91.4** | **86.5** | **82.0** | 91.5 | **86.7** | **82.3** | **88.6** | **87.1** | [1] Universal source-free domain adaptation. In CVPR2020. [2] Exploring generic knowledge and reactivating source model for source-free universal domain adaptation. IEEE TMM, 2026. [3] Pre-trained initialization and memory-enhanced correction for source-free universal domain adaptation, ESWA, 2026.

In this paper, our "unified" refers to (1) the ability to unify multiple known SFDA settings, and (2) our method does not require an explicit setting label or identifier. Additionally, SFDA allows access to unlabeled target data. Thus, the problem setting, such as different category relationships, is already specified by the training data used in each experiment. To avoid ambiguity about the meaning of “unified”, the revised version has removed the statement “without prior knowledge of the target setting or category relationship”.

As a continuation of Part-1, Part-2 is provided below:

−−−−−−−−−−−−−−−−−−−−−−−−−−∗∗Q4∗∗.CouldtheauthorsclarifytheadaptationandevaluationprotocolsforSF−OODG,particularlytheuseofunlabeledtargetsamplesandtheinformationavailabletothecomparedmethods?∗∗R∗∗:Thankyouforrequestingthisclarification.FollowingtheOODevaluationprotocolsofTent[1]andTPT[2],weuseanImageNet−pretrainedResNet−50asthesourcemodelandperformunsupervisedadaptationonImageNet−A,ImageNet−R,ImageNet−Sketch,andImageNet−V2independently.ThetargetclassidentitiesandtheircorrespondencetoImageNetclassesareknown.Themismatchbetweenthesourceandtargetlabelspacesisaddressedbythemaskedclassificationhead.Thedetailshavebeenupdatedintherevisedversion.[1]Tent:Fullytest−timeadaptationbyentropyminimization.InICLR2021[2]Test−timeprompttuningforzero−shotgeneralizationinvision−languagemodels.InNeurIPS2022. -------------------------- **Q4**. Could the authors clarify the adaptation and evaluation protocols for SF-OODG, particularly the use of unlabeled target samples and the information available to the compared methods? **R**: Thank you for requesting this clarification. Following the OOD evaluation protocols of Tent [1] and TPT [2], we use an ImageNet-pretrained ResNet-50 as the source model and perform unsupervised adaptation on ImageNet-A, ImageNet-R, ImageNet-Sketch, and ImageNet-V2 independently. The target class identities and their correspondence to ImageNet classes are known. The mismatch between the source and target label spaces is addressed by the masked classification head. The details have been updated in the revised version. [1] Tent: Fully test-time adaptation by entropy minimization. In ICLR 2021 [2] Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS 2022.

Q5. Could the authors verify the inconsistencies in the reported CLIP results and improvement margins, and clarify the reliability of the relatively small performance gains in some settings?

R: Apologize for these typos. The CLIP results on VisDA in Tab.3 were mistakenly copied from those on OfficeHome and should be the results in Tab.9. The stated improvement of “at least 10.5%” should be “at least 7.0%.” Here, “%” refers to the absolute difference in accuracy, rather than the relative percentage improvement. In the Ethics Statement, “federated learning” should be “unified source-free domain adaptation.” Regarding the relatively small performance gains, we follow SHOT [1] and report results averaged over three runs with different random seeds, which helps mitigate the influence of training randomness.

We have incorporated all changes in the revised manuscript.

[1] Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation, In ICML 2020.

Dear reviewer svuQ,

We sincerely appreciate your thoughtful feedback throughout the review process. Following the initial comments, we have submitted detailed responses to all raised points.

It would be great if the reviewer could kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you again for your time and support!

Best regards, Paper 15511 Authors

Reviewer_zfm8OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses Source-Free Domain Adaptation (SFDA), highlighting the limited practicality of existing methods that primarily focus on specific scenarios, such as closed-set or open-set adaptation. To address this limitation, the authors formulate domain adaptation from a Structural Causal Model (SCM) perspective and propose a learning framework that leverages internal causality (from the target model) and external causality (from a vision-language model). Furthermore, the authors introduce a mutual information-based objective with a theoretical lower bound and empirically evaluate the effectiveness of the proposed framework across various SFDA scenarios.

Critical strengths
  1. The authors identify an important limitation of existing SFDA methods, which primarily focus on specific adaptation scenarios, and provide a well-motivated problem formulation.

  2. The authors construct a tractable variational lower bound based on a Gaussian approximation to the intractable conditional distribution. It is particularly interesting that the learnable covariance is further utilized as a weighting mechanism for pseudo-label generation, connecting the theoretical formulation with the practical framework.

  3. The authors provide extensive experimental evaluations across diverse SFDA scenarios, consistent with their initial motivation. Notably, the inclusion of Generalized SFDA, which requires preserving source-domain performance during target adaptation, further demonstrates the versatility of the proposed framework.

Critical weaknesses
  1. While formulating domain adaptation from a Structural Causal Model (SCM) perspective is reasonable, decomposing complex domain shifts into causal and non-causal factors may be an overly strong simplification. In real-world scenarios, factors such as object shape, texture, and background may interact in complex ways.

  2. Although the proposed covariance-based weighting mechanism provides an intuitive implementation derived from the Gaussian variational lower bound of mutual information, its advantages over existing pseudo-label weighting strategies based on ViL or target model predictions remain unclear.

  3. The implementation details of the learnable diagonal covariance matrices remain unclear, particularly regarding their parameterization, initialization, and optimization. Further clarification would improve the reproducibility of the proposed framework.

Questions for authors
  1. In Equation (5), could the authors clarify whether the direction of the KL divergence is correct? Is this a typographical error, or is it based on a different mathematical derivation? Please correct me if I have misunderstood the formulation.

  2. Equation (6) appears to employ sample-wise diagonal covariance matrices. Are these covariance matrices maintained separately for all samples in the dataset? If so, could the authors clarify the associated memory and computational overhead?

  3. How do the authors ensure the positive definiteness of the learnable diagonal covariance matrices during optimization?

  4. What are the specific advantages of the proposed diagonal covariance-based pseudo-label weighting mechanism over simpler uncertainty-based pseudo-label weighting strategies?

4 comments

Q1. In Eq. (5), could the authors clarify whether the direction of the KL divergence is correct? Is this a typographical error, or is it based on a different mathematical derivation? Please correct me if I have misunderstood the formulation.

R: Thank you very much for pointing out this typo and apologize for this oversight. In Eq. (5), q(Z′∣Z)q(Z'|Z) is variational distribution. If it appears in the numerator as currently written, the KL divergence term should carry a negative sign. We have fixed it.

∗∗Q2∗∗.Equation(6)appearstoemploysample−wisediagonalcovariancematrices.Arethesecovariancematricesmaintainedseparatelyforallsamplesinthedataset?Ifso,couldtheauthorsclarifytheassociatedmemoryandcomputationaloverhead?∗∗R∗∗:Thediagonalcovariancematriceslearnableandsample−specific.Nevertheless,thisdoesnotintroducesubstantialmemoryorcomputationaloverhead.Thisisbecausethemodelistrainedinmini−batches,andthecovariancematrixisappliedtologits,whosedimensionalityequalsthenumberofclasses.Furthermore,thediagonalpropertyallowsustosimplifythecovariancematricestoamatrix.Forexample,onthelargeDomainNet−126dataset,foranybatch,thematrixhaveashapeof∗∗[batchsize=64,numberofclasses=126]∗∗.Thisisnotalargematrixanddoesnotsubstantiallyincreaseresourceconsumption.The∗∗TrainingResourceDemandsAnalysis(seeG.7inAppendix)∗∗haspresentthequantitativeevidence. **Q2**. Equation (6) appears to employ sample-wise diagonal covariance matrices. Are these covariance matrices maintained separately for all samples in the dataset? If so, could the authors clarify the associated memory and computational overhead? **R**: The diagonal covariance matrices learnable and sample-specific. Nevertheless, this does not introduce substantial memory or computational overhead. This is because the model is trained in mini-batches, and the covariance matrix is applied to logits, whose dimensionality equals the number of classes. Furthermore, the diagonal property allows us to simplify the covariance matrices to a matrix. For example, on the large DomainNet-126 dataset, for any batch, the matrix have a shape of **[batch size=64, number of classes=126]**. This is not a large matrix and does not substantially increase resource consumption. The **Training Resource Demands Analysis (see G.7 in Appendix)** has present the quantitative evidence.

Q3. How do the authors ensure the positive definiteness of the learnable diagonal covariance matrices during optimization?

R: In our implementation, we ensure positive diagonal entries using F.softplus(self.b)+self.eps\text{F.softplus(self.b)} + \text{self.eps}, where self.b\text{self.b} is the diagonal covariance matrix, self.eps>0\text{self.eps}>0 and Softplus⁡(x)=1βln⁡(1+exp⁡(βx))\operatorname{Softplus}(x)=\frac{1}{\beta}\ln\left(1+\exp(\beta x)\right).

**Q4**. What are the specific advantages of the proposed diagonal covariance-based pseudo-label weighting mechanism over simpler uncertainty-based pseudo-label weighting strategies? **R**: We appreciate this interesting perspective. We adopt the diagonal covariance-based weighting to satisfy the three properties of $f_w$ described in Lines 308–312. Uncertainty-based weighting does not inherently satisfy the third requirement: filtering unstable domain noise to distill invariant semantics. Specifically, the diagonal covariance matrix is a built-in parameter of $I(Z',Z)$ that inherently captures channel sensitivity, as discussed in Lines 315–318. In contrast, uncertainty does not necessarily reflect domain noise: models may confidently assign incorrect pseudo-labels while remaining uncertain about informative but challenging samples.

Dear reviewer zfm8,

We sincerely appreciate your thoughtful feedback throughout the review process. Following the initial comments, we have submitted detailed responses to all raised points.

It would be great if the reviewer could kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you again for your time and support!

Best regards, Paper 15511 Authors

I appreciate the author's detailed responses regarding the implementation details. The explanations have helped clarify my concerns and improve my understanding of the approach. Therefore, I will maintain my original score.

Thank you very much for your feedback and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

We upload a revised version

We sincerely thank Reviewers zfm8, svuQ, and ykQB for their time and effort. In response to their comments, we revised the manuscript and uploaded the revised version. For a clear view, all updates are highlighted in orange\textcolor{orange}{\textbf{orange}}.

Specifically, the main revisions fall into the following five categories:

  • (1) We have checked all equations in the first-stage design and corrected the sign errors, addressing Q1 from Reviewer zfm8 and Q1 from Reviewer svuQ.

  • (2) To avoid ambiguity, in Sec. Introduction, we have clarified the meaning of “unified” and removed the statement “without prior knowledge of the target setting or category relationship”, addressing Q3 from Reviewer svuQ and Q1 from Reviewer ykQB.

  • (3) We have added results under the open-partial-set setting (see Table 7), addressing Q3 from Reviewer svuQ.

  • (4) We have added the experimental protocols for the SF-OODG setting (see Appendix C.2), addressing Q4 from Reviewer svuQ.

  • (5) We have corrected the typographical errors, addressing Q5 from Reviewer svuQ and Q2 from Reviewer ykQB.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.