acceptodds
Withdrawn from ICLR 2027

Beyond Noisy Proxy: Mitigating the Gap for CLIP Based Source-Free Domain Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Source-free domain adaptation (SFDA) aims to adapt a pretrained source model to an unlabeled target domain without access to source data, aligning with the increasing demand for data security and privacy. Recent advances in vision-language models (VLMs), such as CLIP, provide a promising alternative by exploiting cross-modal knowledge as proxy supervision to facilitate SFDA. However, even with pretrained CLIP to guide SFDA, performance remains sub-optimal. Beyond attributing this to noisy or task-agnostic proxy supervision, we explore a more comprehensive perspective: the modality gap between the image and text embeddings. On the one hand, this gap hinders the CLIP proxy from early and fully exhibiting task-specific distinctions for SFDA. On the other hand, under domain shift, it inevitably induces noisy pseudo-labels from the source model and proxy model. To mitigate these challenges, we propose a unified framework, termed Semantic Cross-modal reAlignment with seLective Pseudo-labeling (SCALP), which facilitates SFDA via improved CLIP proxy supervision and reliable pseudo-label learning. Specifically, we first propose a Class-centroids Distributional Modality Realignment (CDMR) to match textual embeddings with confident visual class-centroids via a training-free affine transformation, so as to mitigate the modality gap. We subsequently induce an Agreement-based Selective Pseudo-label Learning (ASPL) strategy that leverages dual-model agreement to reduce noisy pseudo-labels during adaptation. Extensive experiments demonstrate that SCALP consistently improves adaptation performance across multiple benchmarks, achieving overall state-of-the-art results for SFDA.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Source-free domain adaptation, Modality gap, CLIP

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FnvcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies CLIP based source free domain adaptation. The authors argue that the modality gap between image and text embeddings weakens the quality of CLIP as an external proxy and may further affect pseudo label learning. To address this issue, the paper proposes SCALP, which contains two main parts. CDMR aligns textual class representations with visual class centroids estimated from target data. ASPL then selects supervision according to the agreement between the source model and the CLIP proxy, while using neighborhood consistency for uncertain samples. The method is evaluated on Office 31, Office Home, VisDA 2017, and DomainNet 126. The results show consistent gains over several recent SFDA methods, including DIFO, ProDe, Co Learn++, and DUET. The paper also provides component ablations, prompt studies, backbone studies, and analyses of pseudo label quality and modality distance.

Critical strengths

The paper provides a fairly complete experimental evaluation. It compares with several recent SFDA methods and reports consistent improvements on four standard benchmarks. The ablation study also covers the main components, different CLIP backbones, prompt initialization, and pseudo label quality. Another strength is that the paper considers the modality gap from the perspective of CLIP based SFDA. This is a relevant issue because the quality of the external CLIP proxy can directly affect later adaptation.

Critical weaknesses

1.Limited methodological novelty. CDMR mainly performs class level statistical alignment between textual embeddings and target visual centroids, while ASPL is closely related to DUET in using model agreement and consistency learning. The overall framework therefore appears to adapt and combine existing ideas rather than introduce a clearly new learning principle.

2.The central motivation is not fully verified. The paper argues that sparse textual semantics and the modality gap are important causes of weak CLIP based SFDA. However, the experiments mainly show reduced centroid distance and improved accuracy. Richer class descriptions are not compared, so it remains unclear whether the gain mainly comes from modality realignment or from target specific prototype calibration.

Questions for authors

1.The paper attributes part of the modality gap to sparse text prompts such as “A photo of a cat.” Have you compared such prompts with richer class descriptions that contain more task specific semantics? This would help clarify whether semantic sparsity is really a major cause of the observed gap.

2.CDMR uses target samples assigned by the original CLIP proxy to estimate class centroids. How robust is the method when the initial CLIP predictions are inaccurate? Please provide results under cases with lower initial proxy accuracy or stronger domain shift.

3.ASPL appears related to DUET, since both use agreement between the task model and CLIP to select reliable samples and use consistency learning for uncertain samples. Could you clarify the main conceptual difference beyond the use of neighbor based soft targets?

4.The current modality gap metric is directly based on the distance between textual embeddings and visual class centroids, which is also the quantity optimized by CDMR. Could you provide an additional independent measure of semantic alignment?

5 comments

Response to Q1 (Richer task specific textual context to close the gap).

We thank the reviewer for this insightful suggestion. The connection between textual information sparsity and the modality gap has been systematically investigated by Schrodi et al. 2025. Specifically, their controlled experiments in Fig. 7(a) demonstrate that increasing shared semantic information between images and captions reduces the modality gap within downstream representations. More importantly, Fig. 7(c) further validates this observation on CLIP on real-world CC12M data, where shorter captions consistently produce a larger modality gap. These independent findings provide empirical support for our motivation that cross-modal information imbalance contributes to the modality gap. In our study, although we do not explicitly extract or characterize the aligned textual semantics, Fig. 9 provides indirect intuitive evidence that CDMR enhances cross-modal semantic alignment while closing the modality gap, as reflected by the improved image–text joint semantical similarity. Under the SFDA setting, target images are unlabeled, and reliable target-specific descriptions are generally unavailable. Therefore, CDMR leverages confidence-filtered visual class centroids to incorporate target-specific semantic information into textual class representations, achieving cross-modal realignment without requiring external captions.

To clarify this effect, we are conducting additional experiments on Office-Home under the SFDA setting. To simulate richer textual semantics without paired image–caption, we employ a pretrained (MLLM) to generate task-specific class descriptions in a training-free manner. We then compare these enriched textual representations against standard class-name prompts with respect to the new gap metric

\frac{1}{K}\sum_{k=1}^{K} \frac{1}{N_k} \sum_{i:y_i=k} cos \langle \mathbf z_v^i,\mathbf z_t^k\rangle,$$ where $K$ denotes the total number of classes, and $N_k$ is the number of target samples belonging to class $k$. $\mathbf z_v^i$ represents the visual embedding of the $i$-th target image, while $\mathbf z_t^k$ denotes the textual embedding of class $c$. Here, $y_i$ is the ground-truth label of sample $i$, used exclusively for evaluation. The cosine similarity measures the semantic correspondence between each visual embedding and its corresponding ground-truth textual class anchor. A higher $C_{\mathrm{sim}}$ indicates stronger class-conditional cross-modal alignment. These experiments are currently ongoing, and we will report the results in the revised manuscript. **Table: Effect of MLLM-generated textual descriptions on cross-modal alignment and classification accuracy on Office-Home.** | Textual Representation | Enrichment | Acc. Ar→Cl | Acc. Ar→Pr | Acc. Ar→Rw | Acc. Avg. | $C_{sim}$ Ar→Cl | $C_{sim}$ Ar→Pr | $C_{sim}$ Ar→Rw | $C_{sim}$ Avg. | |:---|:---:|---:|---:|---:|---:|---:|---:|---:|---:| | Customized Prompt | No | 77.2 | 93.5 | 92.9 | 87.0 | 0.23 | 0.27 | 0.27 | 0.25 | | Prompt + MLLM Description | Yes (ongoing) | — | — | 93.6 | — | — | — | 0.31 | — | Schrodi, S., Hoffmann, D. T., Argus, M., Fischer, V., & Brox, T. (2025). Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models. In The Thirteenth International Conference on Learning Representations (ICLR 2025).

Response to Q2 (Robustness to Initial CLIP Prediction Errors)

We acknowledge that confidence-based centroid estimation cannot completely eliminate initial prediction errors, particularly under severe domain shifts. CDMR therefore serves as a coarse, class-level realignment mechanism rather than guaranteeing perfect semantic estimation. To mitigate residual errors, our subsequent ASPL strategy employs dual-model agreement between the independently pretrained source classifier and the CLIP proxy, together with reliability-aware neighbor supervision. This provides a complementary, finer-grained mechanism for constraining erroneous supervision during adaptation.

To further address this concern, we additionally evaluate SCALP on the PACS benchmark (Li et al., 2017), considering all 12 source-free domain adaptation tasks across Art Painting (A), Cartoon (C), Photo (P), and Sketch (S), including challenging transfers from Photo to Sketch, Cartoon, and Art Painting. These experiments provide additional evidence of robustness under substantial visual domain shifts.

Table: Classification accuracy (%) on PACS across 12 source-free domain adaptation tasks. SCALP results are reported as mean ± standard deviation.

MethodA→CA→PA→SC→AC→PC→SP→AP→CP→SS→AS→CS→PAvg.
Source-model55.097.644.661.484.246.768.029.931.220.838.433.650.9
SHOT86.399.539.792.598.771.189.280.950.386.090.599.282.0
TPDS89.599.368.993.299.073.193.184.570.494.193.299.388.1
SCALP99.4±0.0899.5±0.0592.2±0.1798.3±0.1199.6±0.0592.1±0.1898.2±0.1099.4±0.0692.0±0.1898.2±0.1199.2±0.1099.8±0.0297.3±0.10

Performance under challenging Sketch-target domain shifts:

MethodA→SC→SP→SAvg.
Source-model44.646.731.240.8
SHOT39.771.150.353.7
TPDS68.973.170.470.8
SCALP92.292.192.092.1

SCALP achieves an average accuracy of 97.3%, outperforming the strongest competing baseline, TPDS (88.1%), by 9.2 %. Notably, on the challenging adaptation tasks A→S, C→S, and P→S, SCALP achieves improvements of 23.3, 19.0, and 21.6 %, respectively. These results demonstrate the effectiveness of SCALP under substantial domain shifts.

We will explicitly acknowledge the dependence on initial CLIP predictions as a limitation and further clarify the complementary roles of CDMR and ASPL in mitigating error propagation.

Tang, S., Chang, A., Zhang, F. et al. Source-Free Domain Adaptation via Target Prediction Distribution Searching. International Journal of Computer Vision 132, 654–672 (2024).

Li et al., ICCV 2017

Response to Q3 (Distinction from DUET).

As also discussed in our response to Q4 from Reviewer EXUp, the key distinction lies in how supervision is constructed and transferred to uncertain target samples. While DUET primarily relies on dual-model agreement for reliable sample selection and augmentation-based consistency for uncertain samples, our ASPL explicitly exploits the semantic structure of target neighborhoods, incorporating consensus-verified, dual-agreed neighbors into soft supervisory targets for disagreed samples (Lines 275–277). Consequently, uncertain samples are guided by both geometric proximity and dual-agreed semantic guidance, rather than relying solely on augmentation consistency.

Furthermore, ASPL operates in conjunction with CDMR, which performs class-conditional cross-modal realignment to enhance the semantic reliability of the CLIP proxy before selective adaptation. Therefore, our contribution lies not in model agreement itself, but in combining gap-aware semantic realignment with reliable semantic supervision from dual models. We will clarify this conceptual distinction in the revised manuscript.

Reviewer_VAktOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies source-free domain adaptation (SFDA) augmented with pre-trained vision-language models (CLIP) as external proxy supervision. The authors argue that the sub-optimal performance of prior CLIP-based SFDA methods cannot be fully attributed to noisy, task-agnostic proxy supervision alone; rather, a deeper cause is the modality gap between image and text embeddings. To address this, they propose SCALP, a unified framework with two components: (1) Class-centroids Distributional Modality Realignment (CDMR), a training-free affine transformation that aligns semantic-sparse text embeddings with confident visual class-centroids estimated from target images, thereby making the CLIP proxy task-specific before adaptation; and (2) Agreement-based Selective Pseudo-label Learning (ASPL), which assigns one-hot supervision to dual-agreed samples and applies neighbor-based consistency to disagreed samples to reduce noisy pseudo-labels. Experiments on four SFDA benchmarks (Office-Home, VisDA-2017, DomainNet-126, etc.) show consistent gains over the second-best competitor (0.7%–2.4%), establishing new state-of-the-art results.

Critical strengths

Practical and elegant solution. CDMR is training-free and does not require target labels or paired captions, which distinguishes it from post-hoc feature shifting and caption-based contrastive learning that are inapplicable in the SFDA setting. This is a meaningful technical contribution.

Solid empirical validation. The paper reports results on four benchmarks, includes comprehensive ablation studies, and examines key design choices (confidence filtering, prompt customization, fine-tuning strategies, gradient conflict, hyperparameters, computational overhead), which strengthens the credibility of the findings.

Critical weaknesses

Modest margin on some benchmarks. The improvement over the second-best method is 0.7% on one benchmark, which raises the question of statistical significance. I recommend reporting standard deviations over multiple runs or significance tests to substantiate the claimed SOTA status.

Sensitivity of CDMR to initial class-centroid estimation. Since CDMR relies on confidence-filtered visual embeddings to estimate class centroids, its quality depends on the initial proxy's correctness. The authors mention a "confident" filter, but more discussion on how early-stage errors propagate — especially under severe domain shift or class imbalance — would strengthen the analysis.

Generalizability beyond CLIP ViT-L/14. Most experiments use a frozen CLIP ViT-L/14@336 backbone. It would be valuable to clarify whether the conclusions (especially the modality-gap effect) hold for other VLM backbones or resolutions, though the appendix does include some backbone ablations.

Limited discussion of limitations. The conclusion is brief and does not discuss failure cases or scenarios where SCALP might underperform (e.g., extremely sparse classes, open-set categories). Adding an explicit limitations paragraph would improve the paper.

Minor presentation issues. Some figures/notations could be clarified (e.g., the precise definition of "dual-agreement" and the confidence threshold). A few typos and spacing issues (visible in the rendered text) should be corrected.

Questions for authors

Please see the weakness.

4 comments

Response to Q1 (Statistical Significance of Performance Gains)

We thank the reviewer for highlighting the importance of statistical reliability, particularly when the performance margins over the strongest baselines are relatively modest.

As specified in our experimental protocol, all adaptation tasks were independently repeated three times, and the reported accuracies are averaged over the three runs. While representative SFDA methods, including DIFO, DUET, and UC-SFDA, primarily report average performance in their main experimental tables, we agree that reporting standard deviations is important for assessing the stability and reproducibility of our results.

To address this concern, we will include the mean ± standard deviation across three independent runs for all adaptation tasks in the revised appendix, covering different CLIP backbones (ViT-L/14 and ViT-L/14@336). This additional analysis will provide a more comprehensive assessment of the consistency and statistical reliability of SCALP's performance improvements in the revised version.

Table: Classification accuracy (%) and performance stability of SCALP across three independent runs.

MethodBackboneOffice-31Office-HomeVisDA-2017DN-126Avg.
SCALPViT-L/1494.2 ± 0.0690.9 ± 0.1091.4 ± --90.7 ± 0.1691.8± --
SCALPViT-L/14@33695.0 ± 0.0591.0 ± 0.0892.2 ± --91.4 ± --92.4± --

--: Still ongoing.

Response to Q2 (Sensitivity of CDMR to Initial Class-Centroid Estimation)

We agree that reliable centroid estimation is important, particularly under severe domain shifts or class imbalance. Importantly, both class imbalance and initial estimation errors have been considered in our method design.

First, regarding class imbalance, CDMR employs class-wise confidence filtering, selecting samples with above-average confidence within each predicted class for centroid retrieval, rather than applying a global threshold that may disproportionately exclude underrepresented classes. As demonstrated on the imbalanced Office-Home Clipart domain (minority-to-majority sample ratio of 39:132), this strategy preserves centroid candidates across all 65 classes, with an overall selected pseudo-label accuracy of 88.1%, despite the imbalanced class distribution (see Table 11).

Table 11: Comparison of confidence-filtering strategies on Office-Home Clipart.

Selection StrategyAccuracy (%) ↑Classes (#) ↑
Class-wise Mean (Sk\mathcal{S}_k)88.165
No Filter (Entire Xt\mathcal{X}_t)76.165
Fixed Threshold (γ=0.9\gamma=0.9)88.863
Fixed Threshold (γ=0.6\gamma=0.6)87.663
Fixed Threshold (γ=0.5\gamma=0.5)86.764

Second, regarding initial estimation errors, we acknowledge that confidence filtering cannot completely eliminate erroneous predictions under severe domain shifts. We will explicitly discuss this limitation, including the dependence on the general assumption in CLIP-based SFDA that pretrained CLIP provides relatively transferable semantic representations.

To mitigate the propagation of residual errors, our subsequent Agreement-based Selective Pseudo-Label Learning (ASPL) incorporates dual-model agreement between the independently pretrained source classifier and the CLIP proxy, together with reliability-aware neighborhood supervision. Specifically, CDMR provides coarse, class-level noise filtering and cross-modal realignment, while ASPL introduces finer-grained, noise-constrained supervision for target adaptation. These complementary mechanisms are designed to improve robustness against class imbalance and imperfect initial semantic estimation.

Additional evaluation under severe domain shifts. To further address the reviewer's concern, we additionally evaluate SCALP on the PACS benchmark (Li et al., 2017), which features substantial cross-style domain shifts and relatively limited target data. We consider challenging adaptation tasks from the Photo source domain to Sketch, Cartoon, and Art Painting target domains. The results are reported below:

Table 2: Classification accuracy (%) on PACS across 12 source-free domain adaptation tasks. SCALP results are reported as mean ± standard deviation.

MethodA→CA→PA→SC→AC→PC→SP→AP→CP→SS→AS→CS→PAvg.
Source-model55.097.644.661.484.246.768.029.931.220.838.433.650.9
SHOT86.399.539.792.598.771.189.280.950.386.090.599.282.0
TPDS89.599.368.993.299.073.193.184.570.494.193.299.388.1
SCALP99.4±0.0899.5±0.0592.2±0.1798.3±0.1199.6±0.0592.1±0.1898.2±0.1099.4±0.0692.0±0.1898.2±0.1199.2±0.1099.8±0.0297.3±0.10

Performance under challenging Sketch-target domain shifts:

MethodA→SC→SP→SAvg.
Source-model44.646.731.240.8
SHOT39.771.150.353.7
TPDS68.973.170.470.8
SCALP92.292.192.092.1

These results further demonstrate the effectiveness of SCALP under substantial domain shifts. We will incorporate these additional evaluations into the revised manuscript and explicitly acknowledge the limitations associated with imperfect initial CLIP predictions, particularly under severe domain shifts. References

Tang, S., Chang, A., Zhang, F. et al. Source-Free Domain Adaptation via Target Prediction Distribution Searching. International Journal of Computer Vision 132, 654–672 (2024).

Li et al., ICCV 2017

Response to Q3 (Generalizability beyond CLIP ViT-L/14@336)

We clarify that SCALP is not restricted to CLIP ViT-L/14@336. Our experiments in Appendix B additionally evaluate multiple pretrained vision-language backbones, including CLIP ViT-B/16, ViT-B/32, and SigLIP-B/16, with the latter adopting a distinct sigmoid-based contrastive learning objective.

As reported in the additional results below, SCALP consistently improves adaptation performance across different pretrained VLM proxies, achieving up to ** 94.4 % on Office-Home with SigLIP-B/16 **. These findings demonstrate the generalizability of our semantic realignment framework beyond a particular CLIP architecture or contrastive training objective.

The detailed results are presented below and will be included in the revised version:

Table 10: Generalizability of SCALP across different pretrained VLM backbones. Classification accuracy (%) is reported as mean.

MethodBackboneOffice-31Office-HomeVisDA-2017DN-126Avg.
DIFOCLIP ViT-B/1692.282.590.084.887.4
SCALPCLIP ViT-B/1693.386.091.086.889.3
SCALPCLIP ViT-B/3292.585.791.286.388.9
SCALPSigLIP-B/1694.094.492.192.193.2

Response to Q4 (Limited Discussion of Limitations)

We appreciate the reviewer's suggestion and agree that the limitations of our framework warrant explicit discussion. SCALP assumes a known target-class vocabulary and relies on sufficiently reliable initial CLIP predictions for class-centroid estimation and subsequent selective learning.

In particular, severe domain shifts and extreme class imbalance may compromise the reliability of centroid estimation and subsequent supervision. Although CDMR and ASPL are designed to mitigate these challenges, they cannot completely eliminate errors arising from inaccurate initial semantic estimation. Furthermore, our current framework focuses on closed-set SFDA, and extending it to open-set scenarios remains an important direction for future research. We will explicitly acknowledge these limitations in the revised manuscript.

Response to Q5 (Presentation and Notation Clarity)

We thank the reviewer for identifying these issues. We will clarify the definitions of dual-model agreement, class-wise confidence filtering, and the associated selection criteria, while ensuring consistent notation for CLIP proxy predictions and source-model predictions throughout the manuscript.

We will also carefully proofread the manuscript to correct grammatical errors, typographical issues, and inconsistent spacing, thereby improving its overall clarity and readability.

Overall, we sincerely appreciate the reviewer's constructive suggestions, which will help strengthen the experimental reliability, methodological clarity, and discussion of the proposed SCALP framework.

Reviewer_EXUpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of source-free domain adaptation (SFDA) with access to a CLIP proxy model. They mention two challenges: CLIP has a modality gap between the image and text embeddings which makes its use in SFDA difficult, and under domain shift, pseudo-labels from either source model or CLIP will be noisy. To address the first problem, they propose a Class-centroids Distributional Modality Realignment (CDMR) method that matches text embeddings to confident visual class embedding centroids via a training-free affine transformation. For the second problem, they propose an Agreement-based Selective Pseudo-label Learning (ASPL) method that uses agreement between two models to improve pseudo-label quality. Finally, they perform experiments on standard SFDA benchmarks.

Critical strengths
  • The paper achieves state-of-the-art performance on several DA benchmarks.
Critical weaknesses
  • The motivation for the paper indicating 2 challenges: modality gap between image and text encoders of CLIP and noisy pseudo-labels during adaptation. However, a previous paper ReCLIP (Hu et al., WACV 2024) also provides the same two challenges as their motivation and proposes a solution to adapt CLIP itself. This paper cites it, but fails to mention that ReCLIP also poses the same two challenges and similar solutions to it.

    • Specifically, ReCLIP uses a learnable projection on CLIP embeddings to overcome the first problem of modality gap, while this paper uses a similar albeit training-free affine transform on the CLIP embeddings.
    • And for the noisy pseudo-labels, ReCLIP uses agreement between two models to choose high confidence samples, and this paper also uses the same idea of dual model agreement.
    • The only difference is that ReCLIP adapts CLIP itself while this paper adapts another source model instead. But overall, I feel the technical novelty of this paper is quite limited given its similarities to ReCLIP. I suggest that the authors make the similarities and differences with ReCLIP more clear and clarify on the novelty of the paper.
    • The other prior work, DUET (Lee et al., NeurIPS 2025) also uses dual model agreement to overcome noisy pseudo-labels. So novelty w.r.t. this work is also limited.
  • The paper also has a lot of grammatical errors and typos, making it harder to read:

    • “task-agonist” / “template-agonist” / “agonist ambiguity” for agnostic (L69, L429, App. D)
    • “sorely investigating” (L49) and “SFDA sorely offers” (L141) for solely
    • “recasts a task-specific CLIP proxy” (L52) and “recast pretrained CLIP as external supervision” (L43): “recast” doesn’t fit either sentence
    • “correcter adaptation” (App. B.9)
    • “targe -domain samples” (L95), a typo
    • Some broken/incomplete sentences:
      • Lines 42–46: “Nevertheless, even recast pretrained CLIP as external supervision…, the overall performance remains sub-optimal… (Lee et al., 2025) Challenge 1:…” The subject is missing, and the citation runs straight into the challenge list.
      • Lines 58–64: “As image encoders are assumed to retrieve… whereas customized textual prompt… remains informatively sparse.” This is a dependent clause with no main clause.
      • Lines 141–142: “Since SFDA sorely offers target images without any labels or paired captions.” Another fragment.
      • Lines 76–79: “Unlike existing solutions that…, these methods are inapplicable…” The “Unlike” contrasts the wrong things.
    • There's many more (e.g. R-SFDA is listed as CVPR25 in Table 1 but ICCV25 in Table 2).

References

Hu et al., "ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation", WACV 2024

Lee et al., "DUET: Dual-Perspective Pseudo Labeling and Uncertainty-aware Exploration & Exploitation Training for Source-Free Domain Adaptation", NeurIPS 2025

Questions for authors

Please address the weaknesses section.

7 comments

Response to Q1 (Regarding the similarity between CDMR and ReCLIP's modality-gap.)

We thank the reviewer for raising these important concerns regarding the technical distinctions between SCALP, ReCLIP, and DUET. We respectfully clarify that our contributions differ in three aspects: adaptation setting, CLIP proxy refinement, and noise-constrained supervision, rather than merely replacing the adapted model.

As claimed by the reviewer, we clarify that ReCLIP mitigates cross-modal misalignment by learning global subspace projection and subsequent cross-modal self-training, aiming to remove redundant information in CLIP embeddings. In contrast, motivated by cross-modal semantic information imbalance (Schrodi et al. 2025.), our CDMR aims to rebalance cross-modal semantics by compensating the insufficient task-specific discriminative semantics of textual representations relative to visual representations (Lines 62–66).

Specifically, CDMR introduces a training-free and class-wise affine transformation that incorporates confidence-filtered visual class statistics into textual embeddings, enabling semantic-aware distributional realignment rather than global redundancy removal (Lines 80–83).

Importantly, ReCLIP primarily exploits global embedding geometry, whereas CDMR explicitly leverages reliable class-level semantic correspondence for realignment (Lines 176–177). Furthermore, ReCLIP directly adapts CLIP through cross-modal self-training, while SCALP employs CLIP as an external semantic proxy to guide the adaptation of an independently pretrained source classifier.

Thus, the fundamental distinction lies in the underlying motivation and mechanism: global redundancy removal versus class-conditional cross-modal semantic rebalancing.

Table: Classification accuracy (%) on Office-Home using CLIP ViT-L/14.

MethodArClPrRwAvg.
ReCLIP86.175.993.992.087.0
SCALP90.282.196.295.190.90

SCALP achieves an average accuracy of 90.90%, exceeding ReCLIP (87.00%) by 3.90 %, respectively.

These results further support the effectiveness of our gap-aware, class-conditional semantic realignment framework. We will clarify the methodological distinctions from ReCLIP and strengthen the corresponding discussion in the revised manuscript.

References

Hu et al., "ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation", WACV 2024

Schrodi, S., Hoffmann, D. T., Argus, M., Fischer, V., & Brox, T. (2025). Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models. In The Thirteenth International Conference on Learning Representations (ICLR 2025).

Response to Q2 (Regarding the dual-model agreement with ReCLIP)

We acknowledge that ReCLIP employs agreement between two prediction branches, ReCLIP-V and ReCLIP-T. However, both originate from the same pretrained CLIP model, relying on visual embeddings and image–text similarity predictions, respectively. Since the textual encoder alone cannot independently classify images, their agreement is established between two CLIP-derived prediction branches, rather than an external pretrained model in practice.

In contrast, SCALP exploits inter-model consensus between a separately pretrained source classifier and an external CLIP proxy, incorporating complementary representations and predictions. More importantly, unlike ReCLIP's agreement-based high-confidence sample selection, our ASPL further constructs selective supervision for disagreed samples through reliable neighborhood-based semantic propagation (Eq.~(8) and (9), Lines 270--281).

We assign each neighbor with the weight dependent on the agreement by dual models:

wi,j=1N+1[mi,j=1]⋅max⁡kp˙vi,j(k)w^{i,j} = \frac{1}{N} + \mathbf{1}[m^{i,j}=1] \cdot \max_k \dot{p}_v^{i,j}(k)

where 1[mi,j=1]\mathbb{1}[m^{i,j}=1] is an indicator function that equals 1 if the jj-th neighbor of the ii-th image satisfies dual-model agreement, and 0 otherwise. NN denotes the number of retrieved neighbors.

Hence, the distinction is not agreement itself, but how complementary predictions are exploited to constrain noisy supervision. We will note this in the related work of revised manuscript.

Response to Q3 (Regarding the claim that the only difference is the adapted source model)

We respectfully disagree that the distinction is merely the choice of adapted classifier. We are actually from two tracks. ReCLIP directly adapts pretrained CLIP for target-domain classification, whereas SCALP follows conventional SFDA built upon pretrained source models, as in DIFO (Tang et al. 2024), where CLIP serves exclusively as an external supervisory proxy rather than the final classifier.

This difference leads to distinct objectives: ReCLIP refines CLIP itself, while SCALP improves proxy task-specific distinction and supervision reliability to facilitate SFDA. Moreover, noisy predictions from a single model may propagate erroneous semantic relationships, particularly when retrieved neighbors are also unreliable. Incorporating complementary knowledge from an independently pretrained source model enables cross-model consensus to calibrate noisy supervision. This further motivates our dual-model consensus-guided neighbor supervision, rather than relying solely on CLIP-derived predictions as in ReCLIP. We will add the content in the revised section of method and related works.

Regarding optimization, ReCLIP performs cross-modal encoder self-training following its initial projection, whereas SCALP freezes both CLIP encoders and updates only learnable textual prompts. This reduces VLM parameter-update overhead while achieving higher Office-Home adaptation accuracy in our reported comparison (Table 19). Therefore, our contribution extends beyond the choice of adapted classifier to a distinct gap-driven proxy refinement and noise-constrained adaptation framework.

References

Song Tang, Wenxin Su, Mao Ye, and Xiatian Zhu. Source-free domain adaptation with frozenmultimodal foundation model. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.