- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies CLIP based source free domain adaptation. The authors argue that the modality gap between image and text embeddings weakens the quality of CLIP as an external proxy and may further affect pseudo label learning. To address this issue, the paper proposes SCALP, which contains two main parts. CDMR aligns textual class representations with visual class centroids estimated from target data. ASPL then selects supervision according to the agreement between the source model and the CLIP proxy, while using neighborhood consistency for uncertain samples. The method is evaluated on Office 31, Office Home, VisDA 2017, and DomainNet 126. The results show consistent gains over several recent SFDA methods, including DIFO, ProDe, Co Learn++, and DUET. The paper also provides component ablations, prompt studies, backbone studies, and analyses of pseudo label quality and modality distance.
Critical strengths
The paper provides a fairly complete experimental evaluation. It compares with several recent SFDA methods and reports consistent improvements on four standard benchmarks. The ablation study also covers the main components, different CLIP backbones, prompt initialization, and pseudo label quality. Another strength is that the paper considers the modality gap from the perspective of CLIP based SFDA. This is a relevant issue because the quality of the external CLIP proxy can directly affect later adaptation.
Critical weaknesses
1.Limited methodological novelty. CDMR mainly performs class level statistical alignment between textual embeddings and target visual centroids, while ASPL is closely related to DUET in using model agreement and consistency learning. The overall framework therefore appears to adapt and combine existing ideas rather than introduce a clearly new learning principle.
2.The central motivation is not fully verified. The paper argues that sparse textual semantics and the modality gap are important causes of weak CLIP based SFDA. However, the experiments mainly show reduced centroid distance and improved accuracy. Richer class descriptions are not compared, so it remains unclear whether the gain mainly comes from modality realignment or from target specific prototype calibration.
Questions for authors
1.The paper attributes part of the modality gap to sparse text prompts such as “A photo of a cat.” Have you compared such prompts with richer class descriptions that contain more task specific semantics? This would help clarify whether semantic sparsity is really a major cause of the observed gap.
2.CDMR uses target samples assigned by the original CLIP proxy to estimate class centroids. How robust is the method when the initial CLIP predictions are inaccurate? Please provide results under cases with lower initial proxy accuracy or stronger domain shift.
3.ASPL appears related to DUET, since both use agreement between the task model and CLIP to select reliable samples and use consistency learning for uncertain samples. Could you clarify the main conceptual difference beyond the use of neighbor based soft targets?
4.The current modality gap metric is directly based on the distance between textual embeddings and visual class centroids, which is also the quantity optimized by CDMR. Could you provide an additional independent measure of semantic alignment?
5 comments
Response to Q2 (Robustness to Initial CLIP Prediction Errors)
We acknowledge that confidence-based centroid estimation cannot completely eliminate initial prediction errors, particularly under severe domain shifts. CDMR therefore serves as a coarse, class-level realignment mechanism rather than guaranteeing perfect semantic estimation. To mitigate residual errors, our subsequent ASPL strategy employs dual-model agreement between the independently pretrained source classifier and the CLIP proxy, together with reliability-aware neighbor supervision. This provides a complementary, finer-grained mechanism for constraining erroneous supervision during adaptation.
To further address this concern, we additionally evaluate SCALP on the PACS benchmark (Li et al., 2017), considering all 12 source-free domain adaptation tasks across Art Painting (A), Cartoon (C), Photo (P), and Sketch (S), including challenging transfers from Photo to Sketch, Cartoon, and Art Painting. These experiments provide additional evidence of robustness under substantial visual domain shifts.
Table: Classification accuracy (%) on PACS across 12 source-free domain adaptation tasks. SCALP results are reported as mean ± standard deviation.
| Method | A→C | A→P | A→S | C→A | C→P | C→S | P→A | P→C | P→S | S→A | S→C | S→P | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Source-model | 55.0 | 97.6 | 44.6 | 61.4 | 84.2 | 46.7 | 68.0 | 29.9 | 31.2 | 20.8 | 38.4 | 33.6 | 50.9 |
| SHOT | 86.3 | 99.5 | 39.7 | 92.5 | 98.7 | 71.1 | 89.2 | 80.9 | 50.3 | 86.0 | 90.5 | 99.2 | 82.0 |
| TPDS | 89.5 | 99.3 | 68.9 | 93.2 | 99.0 | 73.1 | 93.1 | 84.5 | 70.4 | 94.1 | 93.2 | 99.3 | 88.1 |
| SCALP | 99.4±0.08 | 99.5±0.05 | 92.2±0.17 | 98.3±0.11 | 99.6±0.05 | 92.1±0.18 | 98.2±0.10 | 99.4±0.06 | 92.0±0.18 | 98.2±0.11 | 99.2±0.10 | 99.8±0.02 | 97.3±0.10 |
Performance under challenging Sketch-target domain shifts:
| Method | A→S | C→S | P→S | Avg. |
|---|---|---|---|---|
| Source-model | 44.6 | 46.7 | 31.2 | 40.8 |
| SHOT | 39.7 | 71.1 | 50.3 | 53.7 |
| TPDS | 68.9 | 73.1 | 70.4 | 70.8 |
| SCALP | 92.2 | 92.1 | 92.0 | 92.1 |
SCALP achieves an average accuracy of 97.3%, outperforming the strongest competing baseline, TPDS (88.1%), by 9.2 %. Notably, on the challenging adaptation tasks A→S, C→S, and P→S, SCALP achieves improvements of 23.3, 19.0, and 21.6 %, respectively. These results demonstrate the effectiveness of SCALP under substantial domain shifts.
We will explicitly acknowledge the dependence on initial CLIP predictions as a limitation and further clarify the complementary roles of CDMR and ASPL in mitigating error propagation.
Tang, S., Chang, A., Zhang, F. et al. Source-Free Domain Adaptation via Target Prediction Distribution Searching. International Journal of Computer Vision 132, 654–672 (2024).
Response to Q3 (Distinction from DUET).
As also discussed in our response to Q4 from Reviewer EXUp, the key distinction lies in how supervision is constructed and transferred to uncertain target samples. While DUET primarily relies on dual-model agreement for reliable sample selection and augmentation-based consistency for uncertain samples, our ASPL explicitly exploits the semantic structure of target neighborhoods, incorporating consensus-verified, dual-agreed neighbors into soft supervisory targets for disagreed samples (Lines 275–277). Consequently, uncertain samples are guided by both geometric proximity and dual-agreed semantic guidance, rather than relying solely on augmentation consistency.
Furthermore, ASPL operates in conjunction with CDMR, which performs class-conditional cross-modal realignment to enhance the semantic reliability of the CLIP proxy before selective adaptation. Therefore, our contribution lies not in model agreement itself, but in combining gap-aware semantic realignment with reliable semantic supervision from dual models. We will clarify this conceptual distinction in the revised manuscript.
Response to Q1 (Richer task specific textual context to close the gap).
We thank the reviewer for this insightful suggestion. The connection between textual information sparsity and the modality gap has been systematically investigated by Schrodi et al. 2025. Specifically, their controlled experiments in Fig. 7(a) demonstrate that increasing shared semantic information between images and captions reduces the modality gap within downstream representations. More importantly, Fig. 7(c) further validates this observation on CLIP on real-world CC12M data, where shorter captions consistently produce a larger modality gap. These independent findings provide empirical support for our motivation that cross-modal information imbalance contributes to the modality gap. In our study, although we do not explicitly extract or characterize the aligned textual semantics, Fig. 9 provides indirect intuitive evidence that CDMR enhances cross-modal semantic alignment while closing the modality gap, as reflected by the improved image–text joint semantical similarity. Under the SFDA setting, target images are unlabeled, and reliable target-specific descriptions are generally unavailable. Therefore, CDMR leverages confidence-filtered visual class centroids to incorporate target-specific semantic information into textual class representations, achieving cross-modal realignment without requiring external captions.
To clarify this effect, we are conducting additional experiments on Office-Home under the SFDA setting. To simulate richer textual semantics without paired image–caption, we employ a pretrained (MLLM) to generate task-specific class descriptions in a training-free manner. We then compare these enriched textual representations against standard class-name prompts with respect to the new gap metric
\frac{1}{K}\sum_{k=1}^{K} \frac{1}{N_k} \sum_{i:y_i=k} cos \langle \mathbf z_v^i,\mathbf z_t^k\rangle,$$ where $K$ denotes the total number of classes, and $N_k$ is the number of target samples belonging to class $k$. $\mathbf z_v^i$ represents the visual embedding of the $i$-th target image, while $\mathbf z_t^k$ denotes the textual embedding of class $c$. Here, $y_i$ is the ground-truth label of sample $i$, used exclusively for evaluation. The cosine similarity measures the semantic correspondence between each visual embedding and its corresponding ground-truth textual class anchor. A higher $C_{\mathrm{sim}}$ indicates stronger class-conditional cross-modal alignment. These experiments are currently ongoing, and we will report the results in the revised manuscript. **Table: Effect of MLLM-generated textual descriptions on cross-modal alignment and classification accuracy on Office-Home.** | Textual Representation | Enrichment | Acc. Ar→Cl | Acc. Ar→Pr | Acc. Ar→Rw | Acc. Avg. | $C_{sim}$ Ar→Cl | $C_{sim}$ Ar→Pr | $C_{sim}$ Ar→Rw | $C_{sim}$ Avg. | |:---|:---:|---:|---:|---:|---:|---:|---:|---:|---:| | Customized Prompt | No | 77.2 | 93.5 | 92.9 | 87.0 | 0.23 | 0.27 | 0.27 | 0.25 | | Prompt + MLLM Description | Yes (ongoing) | — | — | 93.6 | — | — | — | 0.31 | — | Schrodi, S., Hoffmann, D. T., Argus, M., Fischer, V., & Brox, T. (2025). Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models. In The Thirteenth International Conference on Learning Representations (ICLR 2025).