acceptodds
Under review as a conference paper at ICLR 2027

CAESAR: Clustering via Autonomous Embedding-Space Agglomerative Reorganization

Abstract

Clustering algorithms that operate on nearest-neighbor graphs, such as FINCH (First Integer Neighbor Clustering Hierarchy), depend heavily on the quality of the embedding space they are given. However, pretrained vision and language model embeddings are not optimized for this purpose. We propose CAESAR, a method that reorganizes a pretrained embedding space: a reorganization network is trained to pull mutual nearest neighbors together and push non-neighbors apart, yielding a reorganized embedding space substantially better suited to clustering. CAESAR offers a second major advantage: it never requires the number of clusters . This matters because in realistic unsupervised settings, is typically unknown and discovering it is often part of the problem, yet most strong clustering methods take it as an input. We therefore design the entire CAESAR pipeline to infer rather than assume it is known. Empirically, reorganizing the embeddings consistently improves clustering over the raw space on both text and image datasets. Since the few deep clustering methods that also infer do not release their code, we complement controlled comparisons with methods that infer on the same embedding space by comparisons with strong deep clustering methods that are given the true , giving them a substantial oracle advantage. Even so, CAESAR outperforms all of them on text, achieves the best results on the most challenging image benchmark and remains competitive on the others. Reorganizing pretrained embeddings thus emerges as a simple and powerful route to clustering realistic data, where classes overlap and the number of clusters is unknown.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
clustering, deep clustering, representation learning, embedding reorganization, contrastive learning, frozen embeddings

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_146bOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAESAR to address two challenges: adapting foundation models to clustering tasks and automatically determining the number of clusters K. Specifically, CAESAR constructs a mutual k-nearest neighbor graph and trains a reorganization network without requiring prior knowledge of the number of clusters. The method is evaluated on three image datasets and one text dataset.

Critical strengths
  1. The method does not require prior knowledge of the number of clusters K.
  2. The method adapts the foundation model to the clustering task and then performs clustering.
Critical weaknesses
  1. Many existing methods have already explored adapting foundation models to clustering tasks and estimating the unknown number of clusters K. Therefore, the problem addressed in this paper is not sufficiently novel.
  2. On some datasets, such as CIFAR-10, directly applying K-means to features extracted from foundation models can already achieve good clustering performance. Although this paper does not require prior knowledge of the number of clusters K, its clustering performance is substantially worse than that of foundation model features + K-means with known K.
  3. The compared clustering methods are not sufficiently recent. The authors should compare their method with the latest clustering methods published at top-tier conferences in 2026. The number of experimental datasets is too limited, and the ablation studies are not sufficiently comprehensive.
Questions for authors

Please see the weaknesses.

Reviewer_MQBSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new Deep Clustering approach geared towards k-estimation. The method extracts the mutual nearest neighbors from the pretrained embedding to use in a low-complexity network to determine the actual embedding. This embedding is then fed into an existing clustering algorithm, FINCH. The new method is validated by experiments on several datasets, though the comparison with competitors is insufficient, leaving significant concerns about the reliability and suitability of the proposed method.

Critical strengths

The most substantial positive aspect is that the goal of the submission (Deep Clustering that is able to determine the cluster number by itself) is useful and the proposed method may be suitable. The usage of a mutual nearest neighborhood as a intermediate state before training the main encoding is interesting.

Critical weaknesses

(W1) There are significant flaws in the submission, specifically with regard to differentation from existing works. The reorganization network ultimately appears not significantly diifferent from a regular deep clustering network. While many make use of very large deep network architectures, in many others (e.g., Guo, et al., "Improved Deep Embedded Clustering with Local Structure Preservation", Leiber, et al., "The DipEncoder: Enforcing Multimodality in Autoencoders") the default architecture is chosen at a similar size. Thus, in the current state, I do not see this as the major contribution the paper makes it out to be. It could be improved if methods using small networks had been used in the evaluation, but the submission does not acknowledge them. There exist significantly more Deep Clustering methods that do not require a preset cluster number than discussed, especially within the subfield of density-based Deep Clustering which generally does not require a specified cluster number. DDC (Ren, "Deep density-based image clustering"), for example, also uses a post-processing with tSNE, which in turn uses neighbor-information, like the proposed method, to get the embedding for clustering. This makes comparison critical.

(W2) The evaluation appears similarly flawed. -- Despite the frequent claim that there are no implementations for Deep Clustering without a preset cluster number to compare against, this is untrue. The ClustPy package features multiple such methods (https://github.com/collinleiber/ClustPy), and the recent SHADE (Beer, et al., SHADE: Deep Density-based Clustering") method also included an implementation (https://github.com/pasiweber/SHADE). In both cases, the training architecture can be swapped. The note regarding the lack of implementations also does not line up with the concrete evaluation format, as the results seem to have been taken from the literature rather than rerun themselves. As such, a lack of usable implementation should not even have been a limitation to inclusion in the submission. A major point of the proposed method is the detection of k for the nearest neighborhood within a pretrained embedding. However, there is no proper examination of this aspect and its reliability. There is no information given with regard to the variance of scores, which is problematic as Deep Clustering methods are generally prone to strong variability in performance across different random states. The text-based success feels overclaimed, as the comparison is only against a single other method besides FINCH (that is part of the proposed method's pipeline) and only on a single dataset. The number of tested datasets (four overall) is also too small to reliably claim a good performance, especially as the proposed method only barely outperforms the second-best method in the one image dataset where it ranks first, and more substantially underperforms in the other two, being outperformed noticeably, including by EVoC which did not receive the ground-truth cluster number.

Questions for authors

None

Reviewer_Uir9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CAESAR learns a transformation of frozen pretrained embeddings to improve clustering without specifying the number of clusters. It constructs a mutual nearest-neighbor graph, trains an MLP with attraction and repulsion losses against a dataset-wide memory bank, and applies FINCH to the transformed embeddings. The final hierarchy level is selected by silhouette score. Experiments cover AG News, CIFAR-10, CIFAR-20, and STL-10, with component ablations and an additional CLIP evaluation.

Critical strengths
  1. A practical problem and a simple pipeline. Improving frozen embeddings without retraining the foundation model is useful, particularly when the number of clusters is unavailable. The method is clearly described and can operate across text and image embeddings.

  2. Consistent improvements over the direct FINCH baseline. CAESAR improves silhouette-selected FINCH on all four datasets. Appendix D also reports substantial improvements with CLIP features, providing evidence that the effect extends beyond the main Qwen encoders. The component ablation supports the importance of including repulsion.

Critical weaknesses
  1. The claimed unsupervised selection protocol is not established. Appendix C compares λ\lambda values using validation NMI, which requires ground-truth labels, and then motivates the choice λ=0.5\lambda=0.5. Sharing this value across datasets avoids dataset-specific tuning but does not establish label-free hyperparameter selection. If this comparison informed the choice, the experimental pipeline uses supervision unavailable in the stated application. The manuscript should distinguish not requiring KK as an explicit input from making all selection decisions without labels.

  2. The methodological contribution is insufficiently distinguished from existing graph-based representation learning. Neighbor attraction, non-neighbor repulsion, neural projections, and memory banks are established components. Parametric UMAP also learns a neural mapping from neighborhood structure Sainburg et al., 2021. The current comparisons do not establish the benefit over simpler embedding transformations followed by the same FINCH procedure. Table 3 demonstrates that pull-only training collapses and that full-bank repulsion improves over in-batch repulsion, but does not isolate mutual-neighbor filtering or automatic neighborhood selection.

  3. The strongest comparative claims exceed the controlled evidence. Table 2 compares CAESAR using Qwen features with published deep-clustering results using ResNet or CLIP. The paper acknowledges this difference, but the competitors’ access to KK does not resolve the encoder and protocol confounds. Moreover, the headline CIFAR-20 advantage over TAC is only 0.40.4 NMI points and 0.20.2 ACC points. Results are averaged over three seeds without reporting variability, leaving the robustness of these small advantages unclear.

  4. Reliable discovery of clustering granularity remains insufficiently validated. Table 8 reports selected cluster counts near the ground truth, but numerical proximity alone does not demonstrate recovery of meaningful partitions. On CIFAR-10 and STL-10, CAESAR achieves ACC of 57.557.5 and 66.866.8, versus EVoC’s 76.076.0 and 82.482.4 on the same features. Appendix B provides useful ARI results, but no per-seed cluster-count variation or hierarchy-level analysis. The automatic neighborhood rule is also underspecified: neither kmax⁡k_{\max}, the subsample size, nor the resulting dataset-specific kk values are reported.

Questions for authors
  1. Selection protocol: Was λ=0.5\lambda=0.5 fixed before inspecting validation NMI? Please state every use of labels in hyperparameter and model selection. If the NMI sweep informed selection, could you report a label-free selection variant or select the shared configuration on separate development data and freeze it for evaluation?

  2. Contribution isolation: On a representative dataset, could you compare PCA/whitening and a neighborhood-preserving transformation such as Parametric UMAP, followed by the same FINCH and silhouette procedure? Please also isolate mutual versus directed neighbors and automatic versus fixed kk, keeping the remaining configuration unchanged.

  3. Controlled comparison and uncertainty: Could you evaluate a representative competing frozen-feature clustering method under the same encoder and data protocol, and report per-seed results or uncertainty for the methods you run? Please separate conclusions supported by matched comparisons from those based on published numbers.

  4. Granularity and reproducibility: Please provide kmax⁡k_{\max}, subsample sizes, selected kk values, and per-seed cluster counts. Could you report cluster counts and NMI/ARI/ACC across the FINCH hierarchy before and after reorganization? A small class-subsetting or imbalance experiment on existing datasets would further test the central unknown-KK claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.