acceptodds
Under review as a conference paper at ICLR 2027

Unsupervised Long-Tailed Adaptation of Vision-Language Models

Abstract

Adapting vision-language models to downstream tasks has achieved remarkable success by leveraging pseudo-labels generated from unlabeled data. Existing methods typically assume a uniform unlabeled data distribution, and thus the resulting pseudo-label distribution is likewise uniform. However, real-world data distributions are often long-tailed. To tackle this, we formalize a new scenario termed Unsupervised Long-Tailed Adaptation (ULTA). Under this scenario, existing methods exhibit a contrasting phenomenon: head-class performance drops sharply, which is distinct from supervised long-tailed learning where tail classes suffer the most. In particular, we uncover that the distributional mismatch not only erodes head-class boundaries, but also pushes head samples into confusable classes, reinforcing the model's inherent bias. To address these issues, we propose a novel model called Margin-Aware Refinement with Structural alignment (MARS). Specifically, we mitigate head-class boundary erosion via Boundary-Preserving Alignment, which takes the zero-shot VLM as a fixed visual reference to suppress probability increases that lack visual support in the training targets. Building upon this, we introduce Margin-aware Self-Refinement, which employs a dynamic adjustment strategy to refine tail and confusable classes while preventing prediction bias. Extensive experiments on nine benchmark datasets demonstrate that MARS outperforms state-of-the-art methods, achieving an average accuracy improvement of 4.71 percentage points. Code is available in the supplementary material.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision-language adaption, Unsupervised Long-tailed Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_j2LkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies unsupervised adaptation of vision-language models when the unlabeled training data are long-tailed. MARS first uses frozen zero-shot visual anchors to correct unsupported changes in pseudo-label probabilities, then refines a student with a mixture of teacher distillation and hard pseudo-label supervision whose weight depends on estimated class frequency and prediction margins. The evaluation covers nine benchmarks at several imbalance ratios, with additional backbone, head/tail, and larger-scale analyses.

Critical strengths

The paper highlights a useful failure mode: adapting to unlabeled long-tailed data can hurt well-recognized head classes when the adaptation procedure pushes predictions toward a uniform class distribution. I like that MARS tries to preserve useful zero-shot structure while still leaving room to correct it, rather than simply balancing pseudo-label counts. The reported gains across nine datasets and several imbalance ratios, together with the head/tail analyses, make this a promising direction for practical VLM adaptation. The component ablations also suggest that both stages contribute to the overall result.

Critical weaknesses
  1. Could you help me pin down which BPA training path produced the reported results? Eq. (5) and Algorithm 1 use weak-view targets to supervise strong views, but the submitted minimal code returns the weak view first and then uses that first view in train_bpa, discarding the strong one. This may just be a difference in the minimal release, so I'm not assuming the tables used that path. Once that is clear, could we compare corrected and uncorrected targets in one existing setting with the same model, views, and training budget? Table 4 shows that the stage helps; this would tell us what the target correction itself adds.

  2. I'm wondering what the MSR gate adds beyond making the teacher's predictions sharper. With the gate fixed, Eq. (10) amounts to learning from a mixture of the teacher's soft prediction and its own top-1 label, so the target still favors the same class even when that label is wrong. Could we keep the BPA teacher and student setup fixed, compare the adaptive gate with a sample-independent gate having the same average weight each epoch, and look at what happens to incorrect teacher labels in rare predicted classes? That seems like a fairly direct way to connect the gains to the proposed handling of noisy supervision.

  3. I got a bit stuck on how the final student handles a new image. In the minimal code, MSR trains and evaluates on cached BPA image features, but the returned student's raw-image path rebuilds visual prompts from text prompts that MSR has updated; freezing the projection weights alone doesn't freeze those visual prompts. Is the intended model the frozen BPA image branch paired with the final student text embeddings? Making that path explicit, and checking that it reproduces the cached-feature logits on one small batch, would help connect the reported evaluation to the new-image prediction in Eq. (11).

Questions for authors

Could you walk me through the three points above: the BPA view path and matched target-correction comparison; what the adaptive MSR gate adds under a matched average weight; and the frozen visual branch used for new-image inference? These are the clarifications that would most affect my current assessment.

2 comments

Dear Reviewer j2Lk,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period ends in 48 hours, we kindly request that you review the rebuttal and confirm whether your concerns have been addressed. Thanks a lot.

Regards from the authors.

Dear Reviewer j2Lk,

Thanks for your time and effort in reviewing this paper. As the author-reviewer discussion period ends in about 24 hours, we would be grateful if you could take a look at our rebuttal and let us know whether your concerns have been addressed. We are happy to clarify any remaining questions before the deadline.

Regards from the authors.

Reviewer_oj9FOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work aims to address the long-tail problem of pseudo labeling based adaptation approach for VLM. The author proposes two modules, Boundary-Preserving Alignment to protect head-class decision boundaries from collapsing and Margin-Aware Self-Refinement to enable the model learning fine-grained visual distinctions for weak, confusable, or tail classes.

Critical strengths
  • strong empirical performance
  • robustness over different model architectures
Critical weaknesses
  • lack of theoretical analysis
  • too many notations, and some of them are not tied with a clear explanation, which harms the readability.
  • the claim is a bit exaggerated on the first study, since the imbalanced label distribution problem is well-studied in DA community, both empirically and theoretically.
  • the so-called uniform pseudo label regularization is introduced by information maximization rather than an assumption regarding data distribution, which has been proved effective in substantial previous works; the author says CPL suffers from this problem, but there is no direct evidence to support the causal relation; I wonder whether CPL's performance get improved if you remove the uniform regularization.
  • the author treat zero-shot CLIP as anchor to refine the probability, the CLIP's pre-training data is already heavily long-tailed and biased.
  • the correction can hardly happens since the condition si,bi>si,cis_{i,b_i} > s_{i,c_i} is not likely to be triggered if zero-shot prediction gives cic_i.
  • despite the learnable visual prompts (θvisual\theta_{\text{visual}}) in image encoder, this changes the geometry of the entire feature space. It does not exactly perform a point-wise geometric push on an existing, extracted vector vv in the manner described by eq (6) and eq (2).
  • the annealing of α\alpha feels heavy engineering; and if the student model is keep updated by the wrong pseudo label from the teacher, how can the margin be guaranteed less than 0; more importantly, the ablation on fixed α\alpha of different value is not provided.
Questions for authors

see weaknesses.

2 comments

Dear Reviewer oj9F,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period ends in 48 hours, we kindly request that you review the rebuttal and confirm whether your concerns have been addressed. Thanks a lot.

Regards from the authors.

Dear Reviewer oj9F,

Thanks for your time and effort in reviewing this paper. As the author-reviewer discussion period ends in about 24 hours, we would be grateful if you could take a look at our rebuttal and let us know whether your concerns have been addressed. We are happy to clarify any remaining questions before the deadline.

Regards from the authors.

Reviewer_K83YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces Unsupervised Long‑Tailed Adaptation, a practical setting for adapting vision‑language models with unlabeled long‑tailed data. The authors find an counter‑intuitive observation: existing pseudo‑label based unsupervised methods damage head‑class performance (opposite to supervised long‑tailed learning), caused by uniform prior assumptions that erode head‑class decision boundaries and amplify confusable‑class bias. A two‑stage approach MARS is proposed under a preserve‑before‑refine principle. Boundary‑Preserving Alignment uses frozen zero‑shot CLIP as visual reference to correct noisy training targets and defend head‑class boundaries. Margin‑aware Self‑Refinement uses a frozen teacher model, combining class frequency prior and sample‑level cumulative margin consistency for dynamic weighting, to improve tail and confusable classes without harming head classes. Extensive experiments across multiple benchmarks show the effectiveness of MARS.

Critical strengths
  1. The paper formalizes the under‑studied ULTA setting. The key empirical finding (head classes degrade most in unsupervised long‑tailed adaptation, opposite to supervised long‑tailed learning) is interesting.
  2. The authors provide well diagnostic analysis.
  3. MARS achieves Strong empirical performance.
Critical weaknesses

The MSR module depends on estimated class frequencies from unlabeled data. Wrong frequency perception caused by poor zero-shot predictions can misadjust the gating weights and hurt the head-tail balance.

Questions for authors
  1. Why the noisy label on tail data can hurt head performance but increase tail performance.
  2. Limited comparison and discussion with long‑tailed active learning.
  3. Why not use some classic long-tailed benchmark, such as ImageNet-LT?
2 comments

Dear Reviewer K83Y,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period ends in 48 hours, we kindly request that you review the rebuttal and confirm whether your concerns have been addressed. Thanks a lot.

Regards from the authors.

Dear Reviewer K83Y,

Thanks for your time and effort in reviewing this paper. As the author-reviewer discussion period ends in about 24 hours, we would be grateful if you could take a look at our rebuttal and let us know whether your concerns have been addressed. We are happy to clarify any remaining questions before the deadline.

Regards from the authors.

Comments on the paper

2 comments

Dear Reviewers,

We are currently in the discussion phase. The authors have responded to the comments, and I would greatly appreciate it if you could take a moment to check the rebuttal and leave any follow-up comments or questions. If the response has addressed your concerns, please also consider updating your review accordingly.

Your feedback is very valuable to both the authors and the decision process. Thank you again for your help.

Best regards, AC

Dear Reviewers,

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period ends in about 3 days, we kindly request that you read the rebuttal and confirm whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.

Regards from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.