acceptodds
Under review as a conference paper at ICLR 2027

STAR: A Spectral Resonance Framework for Training-Free Multi-Domain Learning with CLIP

Abstract

Multi-domain learning (MDL) aims to learn a model that performs reliably across multiple domains. As a powerful vision-language foundation model, CLIP is a promising basis for MDL due to its large-scale pretraining. To better exploit CLIP's pretrained priors, MDL calls for a different design instead of existing heavy architectures and complex training strategies. In this paper, we study multi-domain CLIP features from both spectral and Gaussian geometric perspectives. Our analyses first reveal that dominant class geometry remains domain-dependent, while nuisance-sensitive variations concentrate in spectral-tail directions, providing a reliable basis for self-domain recognition. More importantly, we find that self-domain failures can be complemented by evidence from other domains, and their recoverability is strongly associated with self-domain Gaussian geometry. Based on these findings, we propose SpecTral Adaptive Resonance (STAR), a training-free framework for CLIP-based MDL. STAR first establishes a reliable self-domain base through Subspace Anchor Guidance (SAG), which preserves dominant class-domain spectral structure while suppressing nuisance-sensitive directions. On top of this base, CrOss-domain REctification (CORE) exploits complementary cross-domain spectral evidence under self-domain Gaussian compatibility to enhance self-domain predictions. Extensive experiments show that STAR consistently outperforms previous MDL baselines, and further analyses validate the effectiveness of our design choices. By building reliable self-domain anchors and selectively resonating with complementary cross-domain structures, STAR offers a new geometric perspective on CLIP-based MDL.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multi-Domain Learning, CLIP, Training-Free

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XQTBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies training-free multi-domain learning (MDL) with CLIP. Through spectral and Gaussian-geometry analyses of multi-domain CLIP features, the authors make two observations: (1) nuisance-sensitive variations concentrate in spectral-tail directions while dominant class geometry remains domain-dependent; and (2) residual self-domain failures can be complemented by cross-domain evidence, with recoverability reflected in self-domain Gaussian geometry. Building on these findings, they propose STAR, which consists of Subspace Anchor Guidance (SAG) for establishing a reliable self-domain base and CrOss-domain REctification (CORE) for exploiting complementary cross-domain spectral evidence. Experiments on Office-Home, DomainNet, and FMoW-WILDS show consistent gains over training-based MDL baselines while remaining training-free.

Critical strengths

Clean and well-motivated problem. The framing of CLIP-based MDL as a lightweight feature-space refinement task—rather than heavy architectures or complex optimization—is convincing and timely. The spectral/Gaussian perspective offers a fresh lens distinct from prior parameter-allocation and feature-decomposition methods.

Insightful preliminary analyses. The two geometric findings (spectral-tail concentration of nuisance variation, and cross-domain complementarity gated by Gaussian geometry) are nontrivial and directly justify the two-stage design. The analysis-to-method linkage is unusually tight for this line of work.

Training-free and efficient. Achieving parity or superiority over training-based baselines without back-propagation is practically valuable, and the reported computational cost (on the order of minutes, far below model training) strengthens the claim.

Critical weaknesses

Dependence on the number of retained singular directions (K). The framework appears sensitive to the choice of K, which is addressed only in the appendix. Is K selected per-domain or shared? How robust is STAR to sub-optimal K, and is there a principled or automatic selection criterion? This is central to whether the method is truly "training-free" in practice, since hyperparameter tuning is itself a form of fitting.

Comparison with cache-based training-free methods. Related work discusses Tip-Adapter and the cache-based line of training-free adaptation, yet the main results emphasize training-based MDL baselines. A direct, controlled comparison against strong training-free cache-based methods would clarify whether the gains stem from the spectral geometry itself or simply from a more effective support-set statistics usage.

Statistical significance. Standard deviations are reported in the appendix, but the main tables report mean accuracy only. Given that some improvements may be modest, providing confidence intervals or significance tests for the headline gains would strengthen the conclusions.

Generality of the two findings. The geometric analyses are presented on CLIP features. Are the conclusions expected to transfer to other vision-language backbones (e.g., SigLIP, OpenCLIP variants), or are they specific to CLIP's training? An ablation with an alternative backbone would reinforce generality.

Clarity on "recoverability." The notion of Gaussian-geometry-compatible recoverability is central to CORE, but the paper could define more precisely the criterion used to decide when cross-domain rectification is triggered. A clearer formalization would improve reproducibility.

Questions for authors

Please see the weaknesses.

1 comment
Reviewer_XQTBOpenReview ↗

Thank you for the rebuttal.

The authors have addressed most of the mentioned issues. Some issues still existed, such as all the compared methods are all published before 2023 and no theory grounding is proved. And also, domain shift is a key challenge for multi-domain learning, but no analysis about how to avoid it. The subspace learning is fuzzy, as seen in Fig. 3, the novelty is limited.

Reviewer_D3LAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes STAR, a training-free framework for CLIP-based multi-domain learning. It combines Subspace Anchor Guidance (SAG), which retains dominant spectral directions for each class-domain pair, with Cross-domain Rectification (CORE), which aggregates cross-domain evidence. These scores are fused with CLIP’s text predictions. Experiments on Office-Home, DomainNet, and FMoW under fully supervised and few-shot settings report improvements over zero-shot CLIP and several prompt-learning baselines.

Critical strengths
  1. STAR requires no additional gradient-based training and integrates easily with existing CLIP-based methods.

  2. The paper provides a geometric perspective on multi-domain learning by combining dominant spectral structure with cross-domain rectification, supported by experiments across multiple datasets.

Critical weaknesses
  1. The paper is difficult to follow.
  • In Section 3.1, the RNPS-based class/domain-shift experiments do not clearly specify the reference groups, original samples. Here can you give a example to further specify how to calculate RNPS on reference and original sample in class/domain-shift experiment.
  • In section 3.2, the motivation for MCM and its connection to Gaussian-based reliability weighting should be explained more explicitly when MCM is introduced. Some explanation appears later in Section 4.3, lines 317–319, but the presentation still does not flow smoothly.
  • The sensitivity analysis should explicitly refer to the equations defining its parameters, such as τ\tau in Eq. (7) and α,β\alpha,\beta in Eq. (14), rather than requiring readers to locate these connections themselves.
  1. In the section 3.1, nuisance-sensitive experiment, it compares both head and tail subspace but not in class shift and domain shift experiments. Comparable analyses using tail subspaces and full CLIP features in class shift and domain shift experiments, would clarify whether the observations are specific to dominant directions and strengthen the motivation for the subsequent design.

  2. Pattern III means images for which the self-domain prediction is wrong but at least one other-domain prediction is correct. However, Eq. (12) averages evidence from all other domains, potentially incorporating conflicting evidence from incorrect predictors. Please report how many pattern III cases are correctly classified using the fused CORE scores alone. This would help verify the usefulness of incorporating other-domain evidence to correct self-domain errors.

  3. DomainNet’s average accuracy declines noticeably as τ\tau increases from 0.95 to 0.99, followed by a sharper decline at 0.999. This behaviour differs from Office-Home and FMoW. Could the authors explain this difference?

  4. Section 5.3 does not analyse the final fusion weights in Eq. (16). Since these weights directly control the contributions of text, SAG, and CORE, could the authors conduct a sensitivity analysis of these weight hyperparameters?

Questions for authors

See the questions in the weaknesses section.

Reviewer_1HkeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies training-free multi-domain learning (MDL) on frozen CLIP features with known test domains. Authors argue nuisance shifts hurt spectral tails more than top singular vectors of per-(class, domain) features (Fig. 1). They also argue other domains' subspaces can fix self-domain errors when gated by Mahalanobis margin (Fig. 2). They propose STAR, which sums CLIP text logits, SAG (top-τ\tau SVD projection energy), and CORE (other-domain energies weighted by shared-covariance Mahalanobis distance). These three scores are Z-score normalized and combined with val-tuned weights. On DomainNet, Office-Home, and FMoW, STAR raises zero-shot CLIP from 73.11 / 82.91 / 19.97 to 80.50 / 89.78 / 52.83 and improves CoOp/MaPLe baselines.

Critical strengths
  1. Simple, cheap, and consistently additive. STAR needs no backprop and improves average accuracy over every zero-shot and CoOp/MaPLe model in Tab. 1, 2, 5, and 6.
  2. Large gain under geographic shift. On full-supervised FMoW, STAR raises zero-shot CLIP accuracy from 19.97 to 52.83 (Tab. 1).
  3. Broad experimental grid. Experiments cover 5 supervision regimes, 2 prompt learners (CoOp, MaPLe) across 4 MDL architectures, and rank-selection ablations.
Critical weaknesses

STAR improves zero-shot CLIP and CoOp/MaPLe baselines, but it is a supervised classifier on frozen features (per-class SVD plus shared-covariance Gaussian scoring). Yet paper never compares it against existing training-free CLIP classifiers, and table results seem not supporting its cross-domain mechanism claims.

W1. Missing training-free CLIP and simple feature-classifier baselines (Sec. 2.2, Tab. 1). STAR uses labeled training data (or 16 shots per class-domain). So natural baselines are label-using, training-free CLIP adapters. Yet paper runs none of the methods reviewed in Sec. 2.2, e.g., Tip-Adapter [R1], APE [R2], ProKeR [R3], GDA-CLIP [R4], or TIMO [R5]. It also omits basic feature classifiers (linear probe, nearest-class-mean, per-domain LDA, pooled subspace). Instead, it only compares with zero-shot CLIP (no labels) and self-trained prompt learners (fixed lr 1e-3, 10 epochs). In fact, CORE gate (Eq. 13, shared-covariance Mahalanobis score) is the exact Gaussian classifier used in GDA-CLIP [R4]. I think can not claim that STAR beats prior MDL baselines w/o abovementioned comparisons.

W2. Inconsistent SAG-only scores and weak cross-domain ablation leave CORE's gain unclear (Tab. 3, Tab. 12, Fig. 4). Three places report SAG w/o CORE at same thresholds with conflicting scores on Office-Home / DomainNet / FMoW. Tab. 3 (text + SAG) gives 88.68 / 79.87 / 45.50. Tab. 12 (SAG branch) gives 89.59 / 78.89 / 52.42. Fig. 4 (SAG alone, best τ\tau) shows ~86 / ~72 / ~59-60 (below zero-shot 73.11 on DomainNet, but above full STAR 52.83 on FMoW).

  1. If Tab. 12 is correct, text + CORE add only +0.19 / +1.61 / +0.41 over SAG alone. Meanwhile, the text branch hurts FMoW accuracy by ~7 pts.
  2. Even in Tab. 3, adding cross-domain evidence (CDE) to SAG changes accuracy by only -0.04 / +0.04 / +0.15. Most of the gain comes from the Gaussian-compatibility (GC) term (+5.40 on FMoW). CDE helps only after adding GC (+1.03 / +0.62 / +1.93). So "cross-domain complementarity" is a small effect on top of a Gaussian classifier.

W3. Motivating spectral and domain-specificity analyses lack null controls (Fig. 1, Tab. 13). In Fig. 1, RNPS divides by each subspace's mean reference energy. This denominator is very small in the spectral tail, so large relative tail changes can be an artifact of division. Paper tests no null control (e.g., isotropic noise or random subspaces of same dimension), and does not check class or domain shifts in the tail. Also, paper claims top singular vectors are domain-specific, but never tests a pooled domain-agnostic subspace, and adaptive rank Kc,dK_{c,d} simply grows with sample count per class-domain (Tab. 13).

W4. Unfair validation tuning, non-standard FMoW split, and impossible stds (Tab. 3, Tab. 7-8). Five hyperparameters (τ,α,β,ωSAG,ωCORE\tau, \alpha, \beta, \omega_{\text{SAG}}, \omega_{\text{CORE}}) are tuned on a large labeled validation split, even in few-shot and "unsupervised" settings. Baselines get no such tuning.

  1. FMoW (where STAR gains most) is split randomly 8:1:1 instead of official WILDS time/location-disjoint splits [R6]. Random splitting leaks images of the same satellite site into both train and test.
  2. In Tab. 7, Office-Home per-domain stds (0.01-0.04) are smaller than the minimum possible nonzero std over 3 runs on 265-470 test images (~0.10). Also, Tab. 3 stds do not match Tab. 7-8, and deterministic zero-shot CLIP has nonzero stds.

References

[R1] R. Zhang, W. Zhang, R. Fang, et al. Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification. ECCV 2022. arXiv:2207.09519.

[R2] X. Zhu, R. Zhang, B. He, et al. Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement (APE). ICCV 2023. arXiv:2304.01195.

[R3] Y. Bendou, A. Ouasfi, V. Gripon, et al. ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models. CVPR 2025. arXiv:2501.11175.

[R4] Z. Wang, J. Liang, L. Sheng, et al. A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (GDA-CLIP). ICLR 2024. arXiv:2402.04087.

[R5] Y. Li, J. Guo, L. Qi, et al. Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIP (TIMO). AAAI 2025. arXiv:2412.11375.

[R6] P. W. Koh, S. Sagawa, H. Marklund, et al. WILDS: A Benchmark of in-the-Wild Distribution Shifts. ICML 2021. arXiv:2012.07421.

Questions for authors
  1. Missing baselines (W1, W3). Using the same support data and domain labels, how does STAR compare with training-free CLIP baselines (Tip-Adapter, GDA-CLIP, APE, ProKeR), linear probe / per-domain LDA, and a pooled domain-agnostic subspace?
  2. SAG-only scores & Gaussian control (W2). Which SAG-only numbers are right (Tab. 3, Tab. 12, or Fig. 4)? What does a "text + Gaussian term (GC) only" row score in Tab. 3?
  3. FMoW split & stds (W4). What is STAR's accuracy on official WILDS FMoW splits? What changes across the 3 runs, and how did you get 0.01-0.04 stds on fixed Office-Home test sets (and nonzero stds for zero-shot CLIP)?

Comments on the paper

1 comment

Thank you to the reviewer who has already followed up on the authors' responses. For reviewers who have not yet done so, please post a brief follow-up acknowledging the authors' responses within the next 24 hours, so the authors have time to respond before discussion closes. In your reply, please indicate which main concerns have been addressed, what remains, and whether your overall assessment changes or stays the same.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.