acceptodds
Under review as a conference paper at ICLR 2027

: Benchmarking Image Clustering in the Era of Pre-trained Vision–Language Models

Abstract

Vision–language pre-training has reshaped image clustering, giving rise to language-assisted image clustering (LaIC), which leverages textual semantics to complement visual representations. Despite the rapid proliferation of LaIC methods, it remains unclear how much LaIC has actually advanced image clustering, as existing studies generally suffer from major limitations, including inconsistent experimental settings, inadequate dataset selection, and limited evaluation dimensions. To address this gap, we introduce VLM4Cluster, a comprehensive benchmark for image clustering in the era of pre-trained vision–language models (VLMs). VLM4Cluster implements 17 representative methods spanning classical, deep, and language-assisted image clustering, and evaluates them on 20 datasets covering classical, challenging, fine-grained, large-scale, and out-of-distribution settings. Beyond effectiveness, VLM4Cluster systematically investigates image clustering along three complementary dimensions: robustness to adversarial perturbations, generalization under distribution shifts, and computational efficiency. Our study shows that LaIC substantially advances the clustering performance frontier on many semantically demanding benchmarks, generally exhibits stronger generalization under distribution shifts, and achieves a more favorable effectiveness–efficiency trade-off. However, its gains become less consistent on large-scale and fine-grained datasets, while language assistance does not systematically reduce sensitivity to adversarial perturbations. VLM4Cluster is anonymously released at https://anonymous.4open.science/r/vlm4cluster-iclr2027-c3af98/.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Image Clustering, Vision–Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SLA4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a systematic benchmark of image clustering methods that leverage external textual knowledge through vision-language models. It evaluates 17 methods across 20 datasets from four perspectives: clustering effectiveness, robustness to adversarial perturbations, generalization under distribution shifts, and computational efficiency.

Critical strengths
  1. The paper comprehensively reproduces 17 representative image clustering methods, demonstrating substantial engineering effort.

  2. The benchmark evaluates clustering methods across 20 diverse datasets, providing broad experimental coverage.

  3. The paper systematically evaluates clustering methods from four perspectives: effectiveness, adversarial robustness, generalization under distribution shifts, and computational efficiency.

Critical weaknesses
  1. The novelty of the benchmark is relatively limited. Many recent language-assisted image clustering (LaIC) methods already adopt the same pretrained backbones, such as CLIP ViT-B/32, and evaluate on widely used datasets using standard metrics (ACC, NMI, and ARI). Moreover, recently published LaIC papers have already included comparisons with the most competitive relevant methods. Therefore, the additional contributions of the proposed unified evaluation framework over existing experimental practices are not sufficiently clear.

  2. Although the paper provides extensive experimental comparisons across four evaluation dimensions, the analysis lacks sufficient depth and actionable insights. Most conclusions remain descriptive observations rather than revealing the underlying mechanisms behind the performance differences among clustering methods.

  3. The benchmark does not faithfully reproduce all methods under their original experimental settings. For example, SAC and MAGIC replace their original MLLM-generated textual representations with TAC-style WordNet features, while the default SEIC implementation omits its second-stage optimization. Such modifications may substantially change the original methods and limit the benchmark's practical value as a reusable implementation platform.

  4. The paper observes that language-assisted clustering provides less consistent improvements on large-scale and fine-grained datasets, but does not sufficiently investigate the underlying reasons. For example, why does NTK-SC consistently outperform other language-assisted methods on fine-grained datasets, and why does language assistance improve performance on Flowers and Pets but underperform vision-only methods on Cars and Food? More controlled experiments and detailed analyses are needed to explain these differences and provide meaningful guidance for future research.

Questions for authors
  1. The reproduced clustering results differ considerably from those reported in some original papers, particularly for recent methods such as SAC and SEIC. However, the paper does not provide systematic comparisons with the original results or investigate whether these discrepancies arise from different pretrained models, experimental settings, or implementation modifications. This raises concerns about the fidelity and reliability of the benchmark.

  2. The hyperparameter selection protocol is insufficiently specified. Although the authors provide extensive grid search spaces in Table 8, they do not clarify which criterion is used to select the optimal hyperparameters or whether ground-truth labels are involved in this process. Since the benchmark evaluates unsupervised clustering methods, using label-dependent metrics such as ACC or NMI for hyperparameter selection would compromise the unsupervised evaluation protocol. Furthermore, it is unclear whether different methods receive comparable hyperparameter tuning budgets and whether the computational cost of grid search is included in the efficiency analysis.

  3. The computational efficiency analysis is limited to the total running time on ImageNet-1K, without evaluating computational costs across datasets of different scales. Furthermore, the paper does not provide a detailed breakdown of feature extraction, textual feature construction, clustering training, and inference time, making it difficult to identify the main sources of computational overhead.

  4. Although the authors conduct 10 independent runs, the main results report only average clustering performance without standard deviations, confidence intervals, or statistical significance tests. This is particularly concerning because several conclusions rely on marginal performance differences, such as the 1.1-point ACC improvement on ImageNet-1K and the 0.4-point difference on Places-365.

10 comments

We thank reviewer SLA4 for your valuable comments. We updated the submission accordingly. Please kindly find the detailed responses below.

1. Many recent LaIC methods already adopt the same pretrained backbones, evaluate on widely used datasets using standard metrics, and included comparisons with the most competitive relevant methods.

We highlight that the central contribution of VLM4Cluster is to establish whether the advantages of language-assisted clustering persist when strong vision-only alternatives are evaluated under controlled conditions, and where those advantages break down. Sharing backbones, datasets, and metrics within individual LaIC studies does not resolve this question. Our benchmark addresses three gaps that directly affect the conclusions drawn about progress in image clustering.

  • Including competitive baselines does not ensure a controlled comparison. Although many LaIC methods use CLIP ViT-B/32, comparisons with deep image clustering often reuse results obtained with different network backbones, dataset splits, or image resolutions. Consequently, the reported performance gaps can reflect both methodological differences and experimental choices. VLM4Cluster reproduces 17 classical, deep, and language-assisted methods within a unified codebase, using shared VLM representations, explicit split protocols, and a common image resolution. This allows strong vision-only methods to benefit from the same pretrained representations used by LaIC. Moreover, we examine whether the resulting conclusions persist across ViT-B/32, ViT-B/16, and ViT-L/14 architectures, LAION-400M and LAION-2B pretraining scales, and CLIP and SigLIP objectives (Section 2.3 and Appendix F). A comparison under one commonly adopted backbone cannot establish this generality.

  • Challenging datasets and comprehensive comparisons across paradigms need to be evaluated together. Deep clustering studies commonly emphasize classical datasets, while recent LaIC evaluations on harder datasets primarily compare LaIC methods. This leaves a substantive question unresolved: how much does language assistance improve over strong vision-only clustering in these demanding settings? Our controlled comparisons reveal that the answer changes substantially across datasets. With CLIP ViT-B/32 pretrained on LAION-400M, the best LaIC ACC exceeds the best vision-only result by 15.2 percentage points on ImageNet-Dogs, yet falls 5.3 points below TEMI on Food (Tables 2 and 5). Gains also become substantially smaller on ImageNet-1K. These findings qualify the apparent superiority of LaIC: strong performance on familiar benchmarks does not establish a consistent advantage in large-scale or fine-grained clustering.

  • Standardized metrics do not imply comprehensive evaluation. Existing studies predominantly report ACC, NMI, and ARI on clean or in-distribution data, assessing clustering effectiveness while leaving adversarial robustness, generalization under distribution shifts, and computational efficiency underexplored. VLM4Cluster addresses these gaps through controlled adversarial testing, evaluation across five OOD datasets, and measurement of total pipeline runtime (Sections 3.3–3.5). Crucially, these evaluations distinguish claims that clean-data rankings cannot separate: LaIC generally achieves higher post-attack scores, but does not consistently exhibit smaller relative performance drops. Its higher absolute performance therefore does not establish reduced adversarial sensitivity. Identifying such distinctions is a substantive empirical contribution, because they change what can reasonably be concluded about the benefits and limitations of language assistance.

2. Although the paper provides extensive experimental comparisons across four evaluation dimensions, the analysis lacks sufficient depth and actionable insights.

We kindly note that the specific implications for method development and evaluation derived from these analyses have already been proposed in Appendix I.1.

  • The effectiveness analysis identifies where language assistance breaks down and what needs to improve. The best LaIC result improves ACC over the strongest vision-only baseline by 15.2 points on ImageNet-Dogs, but only 1.1 points on ImageNet-1K, and falls behind by 5.3 points on Food. These comparisons establish that gains on conventional benchmarks do not transfer uniformly to large-scale and fine-grained settings. Section 3.2 identifies a potential mismatch between high-level textual semantics and subtle visual distinctions. Accordingly, Appendix I.1 proposes combining detailed visual cues with more discriminative linguistic knowledge, including dynamically generated descriptions.

  • The robustness analysis distinguishes improved performance from reduced vulnerability. Language assistance generally improves absolute post-attack performance without systematically reducing relative degradation. This distinction prevents higher attacked accuracy from being misinterpreted as stronger adversarial resistance. The resulting research direction is explicit: preserve semantic consistency and stable cluster assignments under perturbations, rather than assume that adding language provides such stability.

  • The generalization analysis accounts for in-distribution performance and identifies shift-dependent strengths. Effective robustness evaluates OOD performance relative to that expected from ID performance. Six of eight LaIC methods achieve positive effective robustness on at least two metrics, yet the leading method varies across shifts. These findings motivate retaining transferable semantic information while reducing reliance on distribution-specific cues, and evaluating generalization across diverse shifts.

  • The efficiency analysis provides concrete guidance for computational trade-offs. TAC++ achieves 77.8% NMI on ImageNet-1K in 66.1 minutes, whereas TEMI achieves 78.1% in approximately 370 minutes. This quantifies the substantial computational premium for a marginal improvement and motivates scalable semantic integration that preserves clustering quality as category diversity increases.

3. Most conclusions remain descriptive observations rather than revealing the underlying mechanisms behind the performance differences among clustering methods.

Indeed, our evaluation does more than establish performance rankings: it distinguishes several competing explanations for the observed differences under a common VLM backbone.

  • Better semantic clustering cannot be explained simply by greater geometric compactness. On ImageNet-Dogs, NTK-SC achieves 80.9% ACC compared with 60.3% for k-means (Table 2), yet k-means achieves better scores on all three internal clustering metrics—SIL, DBI, and CHI (Table 30). Thus, a substantial improvement in recovering semantic categories occurs without improved compactness or separation in the original visual feature space. This exposes an important limitation of geometry-based explanations: visually compact partitions need not correspond to the desired semantic categories.

  • Stronger OOD performance is not simply inherited from stronger in-distribution performance. NTK-SC has lower ImageNet-1K ACC than TEMI—51.9% versus 53.2%—but substantially higher average OOD ACC—38.5% versus 33.8% (Tables 4 and 6). This ranking reversal, together with our effective-robustness analysis, demonstrates that in-distribution clustering quality alone cannot explain the generalization difference. The benchmark therefore distinguishes transfer behavior from improvements in fitting the original distribution.

  • Higher post-attack performance does not establish reduced perturbation sensitivity. TAC++ achieves higher post-attack ACC than TEMI, 48.7% versus 47.3%, despite a slightly larger relative decline, 27.3% versus 26.9% (Table 25). In this comparison, the post-attack advantage reflects stronger clean performance rather than reduced relative degradation. Reporting only attacked accuracy would obscure this distinction and support an incorrect robustness interpretation.

Recalling that our benchmark’s contribution is to establish empirical performance boundaries and identify specific, testable directions for addressing the limitations revealed by controlled evaluation, we believe that detailed mechanistic attribution of performance differences to individual algorithmic components is a separate research objective beyond the scope of this benchmark.

4. Why does NTK-SC consistently outperform other language-assisted methods on fine-grained datasets

NTK-SC’s distinguishing feature is how it incorporates language into the clustering geometry. Appendix B.3.1 provides a concrete mechanism for interpreting this advantage.

Specifically, Eqs. (21–22) give

=\tau^{-2}(z_i^\top z_j)\bigl(p_i^\top p_j\bigr),$$ where $p_i$ is the image’s softmax response over the selected text features. **The affinity therefore combines visual similarity and semantic overlap multiplicatively. Visual similarity remains an explicit factor even when different categories share broad textual descriptors.** Equation (20) further restricts connectivity to mutual neighbors under this joint affinity, and spectral clustering partitions the resulting graph. This construction is particularly relevant to fine-grained recognition: language can reinforce semantic relationships while the visual factor retains distinctions that the vocabulary does not explicitly name. TAC and GradNorm instead apply k-means to concatenated visual and textual representations. NTK-SC therefore supplies a different inductive bias for integrating the same type of lexical information. **Its learned combination of prompt-specific affinities in Eq. (23) additionally allows different templates to contribute unequally.** The resulting design lesson is specific: semantic information should refine pairwise visual relationships while preserving the visual distinctions required by the target partition. This mechanism provides a principled explanation for NTK-SC’s strong results and a concrete direction for improving language-assisted fine-grained clustering. ## 5. Why does language assistance improve performance on Flowers and Pets but underperform vision-only methods on Cars and Food? Thank you for highlighting this dataset-dependent behavior. **We argue that the key factor is the alignment between retrieved textual semantics and the target category distinctions.** In our benchmark, textual guidance is constructed from WordNet nouns. For Flowers and Pets, flower- and breed-related concepts can provide useful semantic anchors for separating categories. Cars, however, requires distinctions between makes, models, and years that generic noun-based representations may not adequately resolve. Food presents a different challenge: substantial appearance variation and distinctions involving local texture, ingredients, and preparation can make retrieved concepts relevant to an image without making them discriminative between categories. Consequently, textual guidance can strengthen category structure when its semantics align with the target partition, but introduce ambiguous cross-class affinities when that alignment is weak.
Reviewer_3afkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VLM4Cluster, a benchmark for image clustering with pretrained vision-language models. It implements 17 algorithms spanning classical clustering, deep image clustering, and language-assisted image clustering, and evaluates them on 20 datasets. Beyond clustering effectiveness, it studies adversarial robustness, generalization to distribution shifts, and computational efficiency. The work's primary contribution is the controlled comparative study and codebase.

Critical strengths
  1. The benchmark includes 17 methods and 20 datasets across classical, challenging, fine-grained, large-scale, and OOD regimes. The inclusion of LaIC alongside classical and deep baselines directly addresses a useful comparability gap. Beyond clustering effectiveness, the four research questions cover adversarial perturbations, distribution shifts, and computational efficiency.
  2. The paper acknowledges that LaIC gains are weaker at scale and inconsistent on fine-grained data, and that high post-attack scores do not imply lower attack sensitivity. It also reports dataset-specific OOD variation and identifies limitations around balanced natural-image data and fixed WordNet vocabularies. These conclusions are useful for researchers in this field.
  3. The reproducibility intent and implementation detail are promising. The appendix gives substantial method-level implementation descriptions, and specifies the software/hardware environment, number of seeds, resizing, prompt templates, and hyperparameter search spaces. These are valuable ingredients for an enduring benchmark.
Critical weaknesses
  1. The authors explicitly explain that some deep clustering methods using raw-image augmentations are not directly applicable to the feature-based setup. This is a defensible scope choice, but still a potential limitation of the benchmark.
  2. The 10-seed means are not accompanied by variability estimates. The paper states that each method's results are averaged over 10 independent random seeds, but the tables report only means and no standard deviations, intervals, or paired comparisons. Several cross-method differences are small, including the large-scale comparisons, so readers cannot tell whether marginal rankings are stable. Reporting variability for headline results would improve the reliability of the benchmark conclusions.
  3. The VLMs are pretrained on web-scale LAION data, while many evaluation images are public. The paper reports no deduplication check against those sources, and a contamination effect therefore cannot be inferred. This is a generic limitation of public-data pretrained VLM evaluations, but an overlap check or explicit caveat would help interpret absolute transfer results.
  4. The classical methods see the unlabeled test images during fitting while the train/test methods do not, so their scores answer slightly different access regimes. Could classical methods be adapted to the train/test paradigm in a naive way? For example, by assigning each test sample to the nearest cluster center computed on the training set.
Questions for authors

Please address my concerns (especially points 2,3,4) in the weaknesses section.

Reviewer_gXg8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VLM4Cluster tests 17 deep and language-based image-clustering techniques using pre-trained vision-language models. It checks 15 image datasets and five modified versions of ImageNet. The study looks at how good the clustering's how sensitive it is to attacks how it handles changes in data and how fast it runs. It also compares encoders. All methods use VLM image embeddings. Language help uses a WordNet setup. The main results show that language help usually makes scores better and makes performance after an attack better.. The improvements are not the same for detailed and big datasets. Also it does not always make the attack damage less. Results with changed data show that language-based methods do better on average. Some simple methods also do well with computer power.

Critical strengths
  1. The comparison data is helpful. Compared to ClustPy’s clustering benchmark and NTK-SC’s study with many datasets VLM4Cluster brings traditional, deep and language-based methods into a bigger test. The test uses shared VLM representations, attacks and time measurements. This helps researchers see if the benefits mentioned in papers still work with the same features.

  2. The results show limits of language help. On ImageNet-Dogs, the best language-based ACC is 82.4% compared to 67.2% for the visual method; on Food the numbers are 69.9% versus 75.2% (Tables 2 and 4). Figures 1–2 also show scores after an attack and less damage from the attack. These differences make the findings more useful.

Critical weaknesses
  1. The way data changes are handled is not clear. Section 3.4 trains methods on ImageNet training data. Leaves out traditional methods as transductive. However TAC, GradNorm and NTK-SC end up using k-means or spectral clustering (Appendix B.3.1). Neither the description nor Appendix E says how to handle data that is not seen. It also does not say if vocabulary, connections or clusters are updated for data. This is important because NTK-SC has the results for out-of-sample data. If some methods update their clusters and others do not the comparison mixes ways of working. The paper does not explain this clearly. A clear way to handle this could fix the problem.

  2. The way models are chosen is not well explained. Appendix E lists a lot of options. It does not say what was used to pick the best settings, which data was used or how long the training was. Configurations that use labels are different from those that do not. The paper does not say which was used. Since the rankings are the part of the study this uncertainty affects how much trust can be placed in the results. The average of ten runs is given without showing how much they vary. This makes it hard to see if small differences are real. Explaining how the choices were made what was. The variation from the runs would help without needing to do the experiments again.

Questions for authors
  1. For TAC, GradNorm and NTK-SC which parts are trained on ImageNet training data and how are images that are not seen assigned? Are images used to change the vocabulary, connections or clusters?

  2. Which data split and what goal were used to pick the settings and how long to train? Were labels used to make the choices? Can the settings and the variation from the ten runs be given for the comparisons?

Comments on the paper

1 comment

Dear Chairs and Reviewers,

We sincerely thank you for your time and efforts throughout the reviewing period. We have uploaded the revised version and responded to all the reviewers in detail. We believe that the quality of the paper is improved and the contributions are solid. In particular, we would like to highlight some key materials we added:

  1. Standard deviations for Stable Analysis (Appendix M)
  2. Detailed Breakdown of Efficiency Analysis (Appendix N)
  3. Inductive Clustering Results of Classical Image Clustering Methods (Appendix O)
  4. Hyperparameter Selection Protocol and Implementation Details for Generalization Analysis (Appendix E).
  5. Explicit Caveat of Possible Data Leakage (Appendix I.2).

We understand that reviewers are busy during the response period, we would greatly appreciate it if the reviewers could kindly advise if our responses solve their concerns. If there are any other suggestions/questions, we will try our best to provide satisfactory answers. We are looking forward to any further discussion with the reviewers.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.