- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a systematic benchmark of image clustering methods that leverage external textual knowledge through vision-language models. It evaluates 17 methods across 20 datasets from four perspectives: clustering effectiveness, robustness to adversarial perturbations, generalization under distribution shifts, and computational efficiency.
Critical strengths
-
The paper comprehensively reproduces 17 representative image clustering methods, demonstrating substantial engineering effort.
-
The benchmark evaluates clustering methods across 20 diverse datasets, providing broad experimental coverage.
-
The paper systematically evaluates clustering methods from four perspectives: effectiveness, adversarial robustness, generalization under distribution shifts, and computational efficiency.
Critical weaknesses
-
The novelty of the benchmark is relatively limited. Many recent language-assisted image clustering (LaIC) methods already adopt the same pretrained backbones, such as CLIP ViT-B/32, and evaluate on widely used datasets using standard metrics (ACC, NMI, and ARI). Moreover, recently published LaIC papers have already included comparisons with the most competitive relevant methods. Therefore, the additional contributions of the proposed unified evaluation framework over existing experimental practices are not sufficiently clear.
-
Although the paper provides extensive experimental comparisons across four evaluation dimensions, the analysis lacks sufficient depth and actionable insights. Most conclusions remain descriptive observations rather than revealing the underlying mechanisms behind the performance differences among clustering methods.
-
The benchmark does not faithfully reproduce all methods under their original experimental settings. For example, SAC and MAGIC replace their original MLLM-generated textual representations with TAC-style WordNet features, while the default SEIC implementation omits its second-stage optimization. Such modifications may substantially change the original methods and limit the benchmark's practical value as a reusable implementation platform.
-
The paper observes that language-assisted clustering provides less consistent improvements on large-scale and fine-grained datasets, but does not sufficiently investigate the underlying reasons. For example, why does NTK-SC consistently outperform other language-assisted methods on fine-grained datasets, and why does language assistance improve performance on Flowers and Pets but underperform vision-only methods on Cars and Food? More controlled experiments and detailed analyses are needed to explain these differences and provide meaningful guidance for future research.
Questions for authors
-
The reproduced clustering results differ considerably from those reported in some original papers, particularly for recent methods such as SAC and SEIC. However, the paper does not provide systematic comparisons with the original results or investigate whether these discrepancies arise from different pretrained models, experimental settings, or implementation modifications. This raises concerns about the fidelity and reliability of the benchmark.
-
The hyperparameter selection protocol is insufficiently specified. Although the authors provide extensive grid search spaces in Table 8, they do not clarify which criterion is used to select the optimal hyperparameters or whether ground-truth labels are involved in this process. Since the benchmark evaluates unsupervised clustering methods, using label-dependent metrics such as ACC or NMI for hyperparameter selection would compromise the unsupervised evaluation protocol. Furthermore, it is unclear whether different methods receive comparable hyperparameter tuning budgets and whether the computational cost of grid search is included in the efficiency analysis.
-
The computational efficiency analysis is limited to the total running time on ImageNet-1K, without evaluating computational costs across datasets of different scales. Furthermore, the paper does not provide a detailed breakdown of feature extraction, textual feature construction, clustering training, and inference time, making it difficult to identify the main sources of computational overhead.
-
Although the authors conduct 10 independent runs, the main results report only average clustering performance without standard deviations, confidence intervals, or statistical significance tests. This is particularly concerning because several conclusions rely on marginal performance differences, such as the 1.1-point ACC improvement on ImageNet-1K and the 0.4-point difference on Places-365.
10 comments
2. Although the paper provides extensive experimental comparisons across four evaluation dimensions, the analysis lacks sufficient depth and actionable insights.
We kindly note that the specific implications for method development and evaluation derived from these analyses have already been proposed in Appendix I.1.
-
The effectiveness analysis identifies where language assistance breaks down and what needs to improve. The best LaIC result improves ACC over the strongest vision-only baseline by 15.2 points on ImageNet-Dogs, but only 1.1 points on ImageNet-1K, and falls behind by 5.3 points on Food. These comparisons establish that gains on conventional benchmarks do not transfer uniformly to large-scale and fine-grained settings. Section 3.2 identifies a potential mismatch between high-level textual semantics and subtle visual distinctions. Accordingly, Appendix I.1 proposes combining detailed visual cues with more discriminative linguistic knowledge, including dynamically generated descriptions.
-
The robustness analysis distinguishes improved performance from reduced vulnerability. Language assistance generally improves absolute post-attack performance without systematically reducing relative degradation. This distinction prevents higher attacked accuracy from being misinterpreted as stronger adversarial resistance. The resulting research direction is explicit: preserve semantic consistency and stable cluster assignments under perturbations, rather than assume that adding language provides such stability.
-
The generalization analysis accounts for in-distribution performance and identifies shift-dependent strengths. Effective robustness evaluates OOD performance relative to that expected from ID performance. Six of eight LaIC methods achieve positive effective robustness on at least two metrics, yet the leading method varies across shifts. These findings motivate retaining transferable semantic information while reducing reliance on distribution-specific cues, and evaluating generalization across diverse shifts.
-
The efficiency analysis provides concrete guidance for computational trade-offs. TAC++ achieves 77.8% NMI on ImageNet-1K in 66.1 minutes, whereas TEMI achieves 78.1% in approximately 370 minutes. This quantifies the substantial computational premium for a marginal improvement and motivates scalable semantic integration that preserves clustering quality as category diversity increases.
3. Most conclusions remain descriptive observations rather than revealing the underlying mechanisms behind the performance differences among clustering methods.
Indeed, our evaluation does more than establish performance rankings: it distinguishes several competing explanations for the observed differences under a common VLM backbone.
-
Better semantic clustering cannot be explained simply by greater geometric compactness. On ImageNet-Dogs, NTK-SC achieves 80.9% ACC compared with 60.3% for k-means (Table 2), yet k-means achieves better scores on all three internal clustering metrics—SIL, DBI, and CHI (Table 30). Thus, a substantial improvement in recovering semantic categories occurs without improved compactness or separation in the original visual feature space. This exposes an important limitation of geometry-based explanations: visually compact partitions need not correspond to the desired semantic categories.
-
Stronger OOD performance is not simply inherited from stronger in-distribution performance. NTK-SC has lower ImageNet-1K ACC than TEMI—51.9% versus 53.2%—but substantially higher average OOD ACC—38.5% versus 33.8% (Tables 4 and 6). This ranking reversal, together with our effective-robustness analysis, demonstrates that in-distribution clustering quality alone cannot explain the generalization difference. The benchmark therefore distinguishes transfer behavior from improvements in fitting the original distribution.
-
Higher post-attack performance does not establish reduced perturbation sensitivity. TAC++ achieves higher post-attack ACC than TEMI, 48.7% versus 47.3%, despite a slightly larger relative decline, 27.3% versus 26.9% (Table 25). In this comparison, the post-attack advantage reflects stronger clean performance rather than reduced relative degradation. Reporting only attacked accuracy would obscure this distinction and support an incorrect robustness interpretation.
Recalling that our benchmark’s contribution is to establish empirical performance boundaries and identify specific, testable directions for addressing the limitations revealed by controlled evaluation, we believe that detailed mechanistic attribution of performance differences to individual algorithmic components is a separate research objective beyond the scope of this benchmark.
4. Why does NTK-SC consistently outperform other language-assisted methods on fine-grained datasets
NTK-SC’s distinguishing feature is how it incorporates language into the clustering geometry. Appendix B.3.1 provides a concrete mechanism for interpreting this advantage.
Specifically, Eqs. (21–22) give
=\tau^{-2}(z_i^\top z_j)\bigl(p_i^\top p_j\bigr),$$ where $p_i$ is the image’s softmax response over the selected text features. **The affinity therefore combines visual similarity and semantic overlap multiplicatively. Visual similarity remains an explicit factor even when different categories share broad textual descriptors.** Equation (20) further restricts connectivity to mutual neighbors under this joint affinity, and spectral clustering partitions the resulting graph. This construction is particularly relevant to fine-grained recognition: language can reinforce semantic relationships while the visual factor retains distinctions that the vocabulary does not explicitly name. TAC and GradNorm instead apply k-means to concatenated visual and textual representations. NTK-SC therefore supplies a different inductive bias for integrating the same type of lexical information. **Its learned combination of prompt-specific affinities in Eq. (23) additionally allows different templates to contribute unequally.** The resulting design lesson is specific: semantic information should refine pairwise visual relationships while preserving the visual distinctions required by the target partition. This mechanism provides a principled explanation for NTK-SC’s strong results and a concrete direction for improving language-assisted fine-grained clustering. ## 5. Why does language assistance improve performance on Flowers and Pets but underperform vision-only methods on Cars and Food? Thank you for highlighting this dataset-dependent behavior. **We argue that the key factor is the alignment between retrieved textual semantics and the target category distinctions.** In our benchmark, textual guidance is constructed from WordNet nouns. For Flowers and Pets, flower- and breed-related concepts can provide useful semantic anchors for separating categories. Cars, however, requires distinctions between makes, models, and years that generic noun-based representations may not adequately resolve. Food presents a different challenge: substantial appearance variation and distinctions involving local texture, ingredients, and preparation can make retrieved concepts relevant to an image without making them discriminative between categories. Consequently, textual guidance can strengthen category structure when its semantics align with the target partition, but introduce ambiguous cross-class affinities when that alignment is weak.
We thank reviewer SLA4 for your valuable comments. We updated the submission accordingly. Please kindly find the detailed responses below.
1. Many recent LaIC methods already adopt the same pretrained backbones, evaluate on widely used datasets using standard metrics, and included comparisons with the most competitive relevant methods.
We highlight that the central contribution of VLM4Cluster is to establish whether the advantages of language-assisted clustering persist when strong vision-only alternatives are evaluated under controlled conditions, and where those advantages break down. Sharing backbones, datasets, and metrics within individual LaIC studies does not resolve this question. Our benchmark addresses three gaps that directly affect the conclusions drawn about progress in image clustering.
Including competitive baselines does not ensure a controlled comparison. Although many LaIC methods use CLIP ViT-B/32, comparisons with deep image clustering often reuse results obtained with different network backbones, dataset splits, or image resolutions. Consequently, the reported performance gaps can reflect both methodological differences and experimental choices. VLM4Cluster reproduces 17 classical, deep, and language-assisted methods within a unified codebase, using shared VLM representations, explicit split protocols, and a common image resolution. This allows strong vision-only methods to benefit from the same pretrained representations used by LaIC. Moreover, we examine whether the resulting conclusions persist across ViT-B/32, ViT-B/16, and ViT-L/14 architectures, LAION-400M and LAION-2B pretraining scales, and CLIP and SigLIP objectives (Section 2.3 and Appendix F). A comparison under one commonly adopted backbone cannot establish this generality.
Challenging datasets and comprehensive comparisons across paradigms need to be evaluated together. Deep clustering studies commonly emphasize classical datasets, while recent LaIC evaluations on harder datasets primarily compare LaIC methods. This leaves a substantive question unresolved: how much does language assistance improve over strong vision-only clustering in these demanding settings? Our controlled comparisons reveal that the answer changes substantially across datasets. With CLIP ViT-B/32 pretrained on LAION-400M, the best LaIC ACC exceeds the best vision-only result by 15.2 percentage points on ImageNet-Dogs, yet falls 5.3 points below TEMI on Food (Tables 2 and 5). Gains also become substantially smaller on ImageNet-1K. These findings qualify the apparent superiority of LaIC: strong performance on familiar benchmarks does not establish a consistent advantage in large-scale or fine-grained clustering.
Standardized metrics do not imply comprehensive evaluation. Existing studies predominantly report ACC, NMI, and ARI on clean or in-distribution data, assessing clustering effectiveness while leaving adversarial robustness, generalization under distribution shifts, and computational efficiency underexplored. VLM4Cluster addresses these gaps through controlled adversarial testing, evaluation across five OOD datasets, and measurement of total pipeline runtime (Sections 3.3–3.5). Crucially, these evaluations distinguish claims that clean-data rankings cannot separate: LaIC generally achieves higher post-attack scores, but does not consistently exhibit smaller relative performance drops. Its higher absolute performance therefore does not establish reduced adversarial sensitivity. Identifying such distinctions is a substantive empirical contribution, because they change what can reasonably be concluded about the benefits and limitations of language assistance.