acceptodds
Under review as a conference paper at ICLR 2027

Toward Robust Human-free Image Annotation with Thousands of Candidate Classes

Abstract

Vision-language models (VLMs) can reduce manual image labeling, but annotation becomes difficult when the label space grows, and the target data are imbalanced. Question-answering VLMs are a natural choice, yet their computational cost increases and their annotation accuracy decreases as the candidate vocabulary grows: on ImageNet subsets, accuracy drops from 73.9% at 200 classes to 49.5% at 1,000. Retrieval-based VLMs such as Contrastive Language-Image Pre-training (CLIP) and Sigmoid Loss for Language-Image Pre-training (SigLIP) scale more efficiently. However, standard zero-shot inference still treats each image independently. We therefore ask: *can the unlabeled target pool itself improve annotation without target labels or knowledge of the true class distribution?* Our analysis shows that class names and target images provide complementary information: the class name is a stable semantic *anchor*, while the pool reveals target-specific visual structure and class prevalence. Standard pseudo-label prototyping only partly exploits this signal, because confidence filtering biases the visual representative and uniform balancing fails under class imbalance. We propose *robust prototyping*, a closed-form semantic–visual calibration that assigns images under an estimated class prior, builds a whitened visual head from all assigned images, and fuses it with the semantic anchor. Without target-pool labels, our method improves zero-shot retrieval on all 18 CLIP/SigLIP encoder-dataset pairs, outperforms the label-free prototype baseline on 15 of 18, and achieves the best results among the evaluated training-free methods on the *long-tailed* ImageNet-LT and Places-LT benchmarks, with only seconds of computation after feature encoding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, human-free image annotation, zero-shot learning, label-free adaptation, transductive learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HM4DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors study an important research question: human-free image annotation on a large-scale dataset with thousands of candidate classes and unknown class distributions. The paper proposes a training-free robust prototyping framework that combines prior-aware pseudo-label assignment, whitened visual prototypes, and semantic–visual fusion. Experiments across multiple datasets and CLIP-family models demonstrate consistent improvements over zero-shot classification and competitive training-free adaptation baselines.

Critical strengths
  1. The proposed method is practical and computationally efficient. It requires neither target annotations nor gradient-based fine-tuning, making it suitable for large-scale automatic labeling. Its closed-form design introduces minimal computational overhead after feature extraction while consistently improving zero-shot performance.

  2. The paper provides extensive experiments across multiple datasets, VLM backbones, and challenging settings, including large candidate vocabularies, class imbalance, and missing classes. The comparisons with existing methods and detailed ablations provide convincing empirical support for the proposed approach.

Critical weaknesses
  1. The task formulation looks closely related to existing transductive zero-shot classification and CLIP adaptation methods, such as TransCLIP [1] and InMaP [2]. Although the paper emphasizes large candidate vocabularies and unknown class distributions, it would be helpful to further clarify what fundamentally distinguishes this annotation setting from prior transductive classification settings.
  2. Although the authors motivate the task through autonomous image annotation in the real world, the experiments are conducted on curated classification benchmarks. It remains unclear how well the approach generalizes to realistic image collections containing ambiguous images, distribution shifts, or out-of-distribution objects. More discussion or evaluation of these cases would strengthen the practical claims.

[1]. https://proceedings.neurips.cc/paper_files/paper/2024/file/71d7dbe2652bd4662d29fa269f059db4-Paper-Conference.pdf [2]. https://arxiv.org/pdf/2310.19752

Questions for authors
  1. Since the estimated class prior is derived from potentially biased zero-shot predictions, how sensitive is the method to errors in the estimation, particularly for rare or unseen classes?
  2. The label-free setting transfers the fusion weight from other datasets. How robust is the approach when it is applied to a new domain whose visual characteristics and class distributions are quite different?
Reviewer_nJ2TOpenReview ↗
Rating
2
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how frozen vision-language models can annotate an unlabeled image pool when the candidate vocabulary is large and the target class distribution is unknown and potentially imbalanced. The method first generates initial predictions using image–text similarities, estimates a target class prior from predicted class frequencies, and produces pseudo-labels through Sinkhorn assignment. It then fits a closed-form ridge classification head using all assigned images and fuses the normalized visual head with the textual semantic anchors at the score level.

The method is evaluated on six datasets with three CLIP/SigLIP encoders, with additional experiments covering long-tailed distributions, missing classes, vocabulary expansion, transfer to DINOv2, and computational cost. The configuration that uses no target labels outperforms zero-shot prediction on all 18 encoder–dataset pairs and the standard prototype baseline on 15 of them. The paper also examines how confidence filtering, class priors, and representation anisotropy affect prototype construction.

Critical strengths

1.The pipeline has practical utility, with improvements extending beyond a single dataset.

In Table 2, the configuration that uses no target-pool labels outperforms zero-shot prediction on all 18 encoder–dataset pairs. For example, SigLIP2 improves from 77.07% to 82.24% on TinyImageNet and from 75.05% to 80.28% on SUN397. The long-tailed and missing-class experiments further demonstrate practical utility under the tested imbalanced conditions. Frozen encoders and a closed-form classification head also reduce the adaptation cost after feature extraction.

2.Several controlled experiments provide useful practical insights.

The confidence-filtering experiments include random subsets of matched size and ground-truth-label controls, helping distinguish selection bias, label noise, and sample-size effects. The paper also directly compares using target labels for hyperparameter selection with using them for supervised training and acknowledges that direct supervised training is more effective. These controls and disclosures improve experimental transparency.

Critical weaknesses

1.The additional contribution over directly related methods and the sources of the performance gains are not sufficiently established.

The core pipeline consists of pseudo-label assignment, a visual classifier constructed using second-order statistics, and fusion with the textual classifier. These components are directly related to existing OT adaptation, GDA classifiers, and prototype fusion methods. Although the paper discusses GDA and IOTA, it does not provide experimental comparisons with these two directly relevant methods, leaving its contribution beyond existing approaches unclear.

The ablation under the recommended prior strengthens this concern. In Table 20, with the estimated prior fixed, a whitened head constructed directly from arg-max pseudo-labels and fused with text achieves 74.27% average accuracy, compared with 74.30% for the full method. The assignment step therefore adds only 0.03 percentage points. Since α and λ are still selected using validation labels, this is not a direct comparison of fully label-free configurations. Nevertheless, it shows that the assignment repair emphasized in the main text provides very limited additional benefit on average across these six datasets. Gains remain on SUN397, so the concern is the scope and significance of the contribution, rather than the assignment step being entirely ineffective.

Furthermore, InMaP uses a softmax marginal, the proposed method uses an arg-max marginal, and OTTER uses its own estimation procedure (Appendix I.12 and Table 25). These comparisons evaluate complete pipelines but do not adequately separate the contributions of the prior estimator and the classification head. Since the paper identifies the class prior as central to robustness, the absence of comparisons under an identical prior leaves a key source of the gains unresolved.

I recognize that simple methods and empirical diagnoses can warrant acceptance. My concern is that the paper has not demonstrated sufficiently clear additional value beyond directly related approaches.

2.The geometric and theoretical explanations supporting the core design contain substantive evidential gaps.

First, Table 12 reports the signed mean of inter-class cosine similarities, yet the main text uses these values to describe the whitened heads as nearly orthogonal. A signed mean close to zero can result from cancellation between positive and negative similarities and does not establish pairwise near-orthogonality. Statistics such as the mean absolute cosine, mean squared cosine, and quantiles are needed. This weakens the central explanation that the head improves classification by making class directions nearly orthogonal.

Similarly, Table 14 compares the label error rate with the prototype directional error, 1 − cos ⁡ , and reports a reduction by factors of several tens. These are different error measures, and their ratio cannot directly be interpreted as a multiplicative reduction in label noise. When representations contain a strong shared direction, a small overall angular change also does not necessarily imply a small error in discriminative information.

Second, the proof of Lemma 1 uses a concentration argument for independent bounded samples conditional on the assignment. However, the actual Sinkhorn assignment depends on the entire image pool. Independence of the original samples does not automatically imply conditional independence under a global assignment event. The proof does not address this dependence, primarily affecting the class-mean concentration term and its use in Proposition 3. This should currently be regarded as an unresolved applicability issue rather than proof that the stated rate is necessarily false.

These gaps do not negate the measured accuracy improvements. However, the correspondence between diagnosis and design is presented as an important contribution, and empirical improvements alone do not validate the proposed explanations.

Questions for authors

1.Additional contribution and performance attribution

Could you compare the proposed method with InMaP/OTTER using identical features, an identical prior, and consistent hyperparameter-selection conditions, and include the directly relevant unsupervised GDA variant and IOTA? Please also compare the full method with “arg-max pseudo-labels + ridge head + text fusion” without using target labels, particularly quantifying the additional benefit of assignment under long-tailed and missing-class conditions. This would directly affect my assessment of the significance of the contribution.

2.Geometric explanation and noise dilution

Please report the mean absolute cosine, mean squared cosine, and high quantiles of absolute cosine similarities for the whitened heads, and clarify whether they support the near-orthogonality claim. For pseudo-label and ground-truth-label prototypes, please examine errors after removing the shared direction, or report directly discriminative measures such as classification margins, to support the noise-dilution explanation.

3.Applicability of the theory

How does Lemma 1 account for the sample dependence introduced by global Sinkhorn assignment? Could you provide an appropriate concentration argument or explicitly restrict the result to a simplified model with independent assignments? Please also clarify the implications of such a restriction for the class-mean concentration term in Proposition 3.

4.Rare classes and the limits of robustness

Appendix F.3 states that classes with zero initial prediction frequency receive no assigned images. How are classes that are actually present but have zero initial predicted counts handled? Equation (2) then produces a zero vector; how is the subsequent L2 normalization defined? Please report the final recall of these classes and distinguish genuinely absent classes from initially missed classes. The long-tailed results should also report overall, macro, and tail-class metrics under the fully label-free configuration, since the appendix already indicates that overall improvements do not translate into consistent macro or tail-class improvements.

5.Interpretation and reproducibility of the many-class experiments

The vocabulary-expansion experiments simultaneously increase the number of classes and images and change the class composition, so they do not isolate the effect of the number of candidate classes. Could you keep the target image pool fixed while adding candidate distractor classes? In addition, Appendix B.2 discloses that per-image outputs for the QA curve were not retained, and the batch sizes used for the timing runs were not recorded. Please clarify which accuracy and cost conclusions can be independently checked and restrict precise speedup claims to the measured implementations. This concern carries less weight than the contribution and mechanism issues above.

Reviewer_t9EAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies label-free annotation of an unlabeled image pool against a fixed vocabulary of up to 1,000 classes. It argues that retrieval VLMs suit this task better than QA-style MLLMs. It proposes “robust prototyping”, a closed-form, three-stage procedure.

  • Sinkhorn assignment of zero-shot scores onto an estimated class prior.
  • A ridge-regularized (“whitened”) class-mean head built from all assigned images.
  • Fusion of this head with the text anchor. Across six pools and three CLIP/SigLIP encoders, it improves over zero-shot in all 18 pairs, and it does best among training-free methods on ImageNet-LT and Places-LT.
Critical strengths
  • The diagnosis is careful. Showing that confidence filtering hurts even with perfect labels (selection bias) and that class means are nearly collinear (cos 0.71–0.88) is useful, and each design choice is tied to a controlled ablation.
  • The method is simple, closed-form, and cheap (seconds after encoding).
  • Baselines are run on identical caches and splits, the InMaP reproduction is verified against the official code, and results cover multiple seeds and stress tests.
Critical weaknesses
  • Gains over strong baselines are small in the label-free setting. The label-free version beats the prototype recipe by under 1 point on average (e.g., 80.04 vs. 79.38 for SigLIP2). On the balanced pools it ties or trails InMaP (ViT-B/16: 67.59 vs. 68.44). On the long-tailed pools, InMaP or OTTER with an estimated prior is within about 1 point. Tail-third accuracy also drops 1.7–6.3 points versus a uniform prior, which matters for annotation, where rare classes are usually the hard part.
  • Limited novelty and a missing key baseline. Sinkhorn balancing (InMaP, OTTER, IOTA), Σ⁻¹μ class heads, and text–visual fusion (Tip-Adapter, Frolic) all exist already. The whitened head is essentially the GDA head of Wang et al. (2024), which the paper cites but does not compare against in Table 3.
  • The framing overstates the scope. The title promises “thousands of candidate classes”, but experiments stop at 1,000 curated benchmark classes. The method is closed-set and cannot flag absent or unknown classes. Its “label-free” λ is also tuned on labels from other pools. The QA comparison uses a 2B model with the full 1,000-name list in the prompt, which reads as a weak strawman rather than a strong motivation.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.