- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work on unsupervised image clustering focuses, unlike other works, on how to effectively utilise given representations for this task rather than how to derive useful representations. For this, the paper motivates the existance of underutilised information in representations mathematically and empirically, and proposes a framework, CARE, which aims to improve the clustering performance through a linear representation projection and local neighbourhood consistency. This framework is evaluated on 9 datasets of varying complexity across different representations and results in consistent gains.
Critical strengths
The paper evaluates the proposed method across different representations (Table 1, 2, 3) and reports consistent improvements. This is important as the paper focuses on improving the utilisation of given representations, the results therefore suggest that the proposed method can increase performance for a variety of representations.
The paper provides empirical evidence for the utilisation gap (Figure 3) and analyses the impact of "structural support" (l.417/418), i.e. the reliability of the local structure in representation space w.r.t. ground-truth labels, on the improvement through CARE (Figure 4). The former clearly motivates the introduction of the proposed method CARE which aims at improving the utilisation of given representations. The latter provides insights into the importance of a well-structured representation space for CARE, where samples close in representation space share the same ground-truth label.
Critical weaknesses
The clarity of the setup description could be improved. The paper introduces the clustering model as producing the "soft assignment" (l.259/260) but does not specify its exact architecture. It is later referred to as "the default linear clustering readout" (l.358) but the exact setup (if applicable softmax activation function, temperature, etc.) is not mentioned. The paper mentions that the goal of the "inner learner" (l. 256/257) is to "[expose] information in the frozen representation" (l.257/258) that can be integrated into the "cluster assignment" (l.258/259) but does not specify how the clustering model incorporates this information from the inner learner.
The paper refers to the appendix for any information about ablations or parameter sensitivity (ll.274/275,309,360) which are necessary to understand the impact of the individual contributions. Even just briefly summarising the main ablation results in the main paper would improve this.
Questions for authors
Figure 3 illustrates the learning-utilization gap for different representations, motivating CARE. How much does CARE close this gap, i.e. what is the remaining learning-utilization gap when using CARE?
3 comments
Thank you for addressing my points and for uploading the revised manuscript, I have no further questions at this stage.
We are pleased to learn that our responses addressed your concerns. Thank you for your constructive review and for your role in strengthening our manuscript.
Response to Reviewer cWAG
We are very grateful for your valuable comments and questions. We try our best to address your questions as follows.
To W1 (Clarity of the Setup Description):
Thank you for your valuable comment.
1.CARE employs a single-layer linear clustering readout that maps frozen visual features to clustering logits, followed by a standard Softmax function to produce soft clustering assignments, without introducing an additional temperature parameter.
2.CARE utilizes the information obtained by the Inner Learner through an alternating optimization mechanism. Specifically, the Inner Learner is first trained using the soft clustering assignments generated by the current outer clustering readout. Its predicted distributions are then used as supervision to update the parameters of the outer clustering readout through a cross-entropy loss. Through this alternating process, the class-discriminative information encoded in the frozen representations is progressively transferred into clustering assignments.
We have clarified these details in the revised manuscript.
To W2 (Clarity of Ablation and Parameter Sensitivity Results in the Main Paper):
Thank you for your valuable suggestion. We have moved the ablation studies to Section 5.4 of the revised manuscript's main text and briefly summarized the main results of the parameter sensitivity analysis in Section 5.7 of the main text.
To Q1 (Reduction of the Learning–Utilization Gap and Remaining Gap):
Thank you for your insightful question.
We have conducted further analyses using TAC representations, STL-10 and ImageNet-10 are excluded from the statistics because their initial gaps are extremely small. The original and remaining gaps are defined as kNN oracle accuracy minus TAC and TAC+CARE accuracy, respectively. TAC accuracy is obtained from the original clustering head to measure the overall gap reduction achieved by the complete CARE framework. The results summarized below:
CARE reduces the average Learning–Utilization Gap from 14.1 pp to 10.9 pp, achieving a mean per-dataset gap reduction ratio of 29.8% across 7 datasets. These results demonstrate that CARE can effectively utilize part of the previously underutilized class-related information, although further improvements remain possible. We have incorporated these experiments and analyses into Section 5.3 of the revised manuscript.