acceptodds
Under review as a conference paper at ICLR 2027

Structure Shapes Semantics: A Unified Framework for Text-Assisted Image Clustering

Abstract

Text-assisted Image Clustering (TIC) leverages vision-language models to provide external textual semantic information, thereby enhancing clustering performance. Existing approaches typically follow a two-stage paradigm: semantic representation construction followed by clustering network training. However, these methods heavily rely on handcrafted strategies or point-to-point matching for semantic representation construction, making them vulnerable to imperfect image-text alignment and semantic noise. In this work, we first introduce a principled framework that reformulates semantic representation construction as a global optimization problem to address these challenges, and propose to solve it via coordinate descent. Then, guided by the principle that "tructure hapes emantics" (), we introduce several semantic-quality criteria and formulate them as optimization objectives. Building upon this reformulation, we further unify the two TIC stages under the shared objective and reveal their symmetric optimization processes. To further refine the semantic representations, we propose the Fine-Structure guided Optimization (FSO). Extensive experiments validate the effectiveness of our method, with a 6.0% accuracy gain on the most challenging Tiny-ImageNet. We believe this work advances TIC from empirically driven heuristics toward a principled optimization paradigm. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Clustering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XfizOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes S³, a unified framework for text-assisted image clustering that addresses the challenges posed by imperfect image-text alignment and semantic noise. First, coordinate descent is employed to select K texts from WordNet candidates that yield better clustering predictions. Second, a combination of shared loss functions and the FSO loss is used to optimize image cluster assignments. Experiments on seven datasets demonstrate the effectiveness of the proposed method.

Critical strengths
  1. This paper transforms the traditional two-stage TIC framework into an end-to-end optimization framework.
  2. Experimental results demonstrate that S³ identifies more suitable text features to improve image clustering performance.
  3. The performance improvement on Tiny-ImageNet is particularly significant.
Critical weaknesses
  1. Numerous existing methods, such as SIC, TAC, GradNorm, CAE, SEIC, and DRSC, have already explored how to better leverage WordNet or textual features generated by LLMs to improve image clustering performance. Moreover, the proposed method appears to be primarily a combination of existing components. Therefore, the novelty of the proposed method appears limited.
  2. Compared with existing methods that more thoroughly exploit WordNet, the proposed method does not demonstrate clear advantages in either computational efficiency or clustering performance.
  3. The theoretical analysis does not adequately support the central scientific claims of the paper. Instead, it primarily focuses on the optimization properties of the proposed method.
  4. The paper only reports runtime results on the DTD dataset without comparisons against other closely related methods. Moreover, the reported computational cost on DTD alone is already substantial, with most of the runtime consumed by Stage 2.
  5. The paper reports a remarkably large performance improvement on Tiny-ImageNet, but does not provide a sufficiently convincing analysis of the factors contributing to this improvement.
  6. Could the authors report the clustering performance and runtime of the proposed method on ImageNet-1K?
Questions for authors

Please see the Weaknesses.

Reviewer_JAx1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduce a principled framework that reformulates semantic representation construction as a global optimization problem to mitigate the limitations caused by handcrafted strategies or point-to-point matching during semantic representation construction. To further refine the semantic representations, the authors proposes the Fine-Structure guided Optimization.

Critical strengths
  1. Evaluation metrics are well-round

  2. Related work are informative

  3. Text-Assisted Image Clustering is a promising research problem

  4. the proposed method achieves a 6.0% accuracy gain on the most challenging Tiny-ImageNet.

Critical weaknesses
  1. Important implementation details, e.g., Stage 2 training epoches, training batch size, prompt setting, and neighborhood size, remain unclear.

  2. In section 4.1 and Appendix B, the authors explciitly require that every valid solution must contain exactly KK distinct noun identities. But Eq. (8) and Algorithm 1 update by searching the vocabulary, together with the current coordinate value, without any efforts that exclude nouns currently occupying other coordinates.

  3. Several reported improvements are inconsistent with the displayed numbers. For example, | Comparison | S³ | Comparator | Actual difference | Reported difference | |---|---:|---:|---:|---:| | CIFAR20 NMI versus KEC | 65.6 | 66.4 | −0.8 | +2.8 in Table 2 | | DTD ACC versus GradNorm | 50.6 | 50.9 | −0.3 | +0.5 in Table 2 | | CIFAR10 ACC versus CDC | 96.6 | 94.9 | +1.7 | +3.8 in Section 5.2 |

  4. There lacks connection between Eq. (8) and Eq. (12). In addition, neighborhood consistency along with entropy-based balancing are already proposed by SCAN.

  5. With G=[g(t1),…,g(tK)]G=[g(t_1),\ldots,g(t_K)], it is clear that z~i⊤z~j=pi⊤G⊤Gpj\tilde z_i^\top\tilde z_j=p_i^\top G^\top Gp_j. From this view, FSO is just a contrastive learning on nomalization cosine similairty. it is unclear why perform normalization on cosine similarity can help.

  6. Corollary 6 is thorectically unjustified and over-claimed. In Corollary 6, the author directly jumps from detecting sufficiently large sampled improvements to convergence to a neighborhood of a coordinate-wise local minimum. Given that detecting an improving candidate does not necessarily imply that repeated noisy updates eventually enter and remain in such a neighborhood, Corollary 6 only provides a conditional, single-update approximation guarantee, rather than the convergence claim for Algorithm 1.

  7. the authors lack comparision with sota methods, e.g., NTK-SC [1] and MAGIC [2], and on imagenet-1k, fine-grained datasets. It is unclear how the proposed method robust to domain shift.

[1] delving into spectral clustering with vision-language representations ICLR26

[2] Magic: Multi-granularity language-informed image clustering ICML26

Questions for authors

see weakness

Reviewer_7EnEOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces S3S^3 (Structure Shapes Semantics), an image clustering algorithm that leverages text via vision-language models. The paper criticizes the current practice in the field of having a two-stage setup, where the first stage of learning a semantic text representation relies on heuristics and handcrafted priors. The authors suggest that the semantic text representation can be learned via coordinate descent and fixed image features, which is then used in stage 2 to learn the clustering while keeping the semantics fixed.

Critical strengths

S1. Well-motivated approach based on a detailed reflection on simple heuristicis used in the field.

S2. Extensive comparison across methods and datasets with large improvements on Tiny-ImageNet.

S3. Substantial ablations and evaluation settings.

Critical weaknesses

W1. Assumption of balanced clusters via their balanced clustering loss (Equation 11), might be problematic in real world clustering datasets

W2. Runtime analysis/experiments are insufficient.

Questions for authors

1. How robust is the proposed method to imbalanced clusters and outliers? I would assume that the balanced clustering loss (Equation 11) is biasing the method towards equal cluster sizes, which is true for the benchmark datasets used, but might not hold for real data, e.g. challenging long tailed datasets like iNaturalist.

2. How does the computational efficiency of the method compare to competitors? In Appendix E, only the method itself is analysed. It would help to know the relative computational overhead vs. relevant baselines.

3. As far as I understood, you only conduct each stage once, but given that you have a unified objective with different trainable parts in each stage, you could alternate between the stages to further improve performance. Did you explore this? Is it stable, or does it degrade the clusterings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.