acceptodds
Under review as a conference paper at ICLR 2027

From Learning to Utilization: Better Exploiting External Knowledge for Image Clustering

Abstract

Unsupervised image clustering increasingly leverages external knowledge, including textual vocabularies, LLM-generated descriptions, and spatial priors. However, existing studies mainly focus on how to learn useful external knowledge into image representations, while overlooking whether the learned knowledge is effectively utilized in the final cluster assignments. In this work, we identify a previously underexplored learning-utilization gap: useful category-related knowledge may be encoded in the representation but remain underutilized in clustering predictions. We formalize learning failure, utilization failure, and their gap from an information-theoretic perspective, and further show that distance-based structural relations with correctly predicted neighbors can recover underutilized knowledge and correct erroneous predictions. Guided by this theory, we propose CARE. CARE first establishes stable initial cluster assignments from externally enhanced representations through iterative pseudo-label optimization, then evaluates sample-level knowledge-utilization reliability, and finally preserves reliable predictions while adaptively exploiting local feature-space structure to refine unreliable ones. In this way, CARE progressively better aligns cluster assignments with the category-related knowledge encoded in the learned representation. Extensive experiments on 6 widely used benchmark datasets and 3 more challenging datasets, covering 3 different types of external auxiliary information, demonstrate that CARE improves clustering assignments. The code will be published.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Image Clustering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cWAGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work on unsupervised image clustering focuses, unlike other works, on how to effectively utilise given representations for this task rather than how to derive useful representations. For this, the paper motivates the existance of underutilised information in representations mathematically and empirically, and proposes a framework, CARE, which aims to improve the clustering performance through a linear representation projection and local neighbourhood consistency. This framework is evaluated on 9 datasets of varying complexity across different representations and results in consistent gains.

Critical strengths

The paper evaluates the proposed method across different representations (Table 1, 2, 3) and reports consistent improvements. This is important as the paper focuses on improving the utilisation of given representations, the results therefore suggest that the proposed method can increase performance for a variety of representations.

The paper provides empirical evidence for the utilisation gap (Figure 3) and analyses the impact of "structural support" (l.417/418), i.e. the reliability of the local structure in representation space w.r.t. ground-truth labels, on the improvement through CARE (Figure 4). The former clearly motivates the introduction of the proposed method CARE which aims at improving the utilisation of given representations. The latter provides insights into the importance of a well-structured representation space for CARE, where samples close in representation space share the same ground-truth label.

Critical weaknesses

The clarity of the setup description could be improved. The paper introduces the clustering model as producing the "soft assignment" (l.259/260) but does not specify its exact architecture. It is later referred to as "the default linear clustering readout" (l.358) but the exact setup (if applicable softmax activation function, temperature, etc.) is not mentioned. The paper mentions that the goal of the "inner learner" (l. 256/257) is to "[expose] information in the frozen representation" (l.257/258) that can be integrated into the "cluster assignment" (l.258/259) but does not specify how the clustering model incorporates this information from the inner learner.

The paper refers to the appendix for any information about ablations or parameter sensitivity (ll.274/275,309,360) which are necessary to understand the impact of the individual contributions. Even just briefly summarising the main ablation results in the main paper would improve this.

Questions for authors

Figure 3 illustrates the learning-utilization gap for different representations, motivating CARE. How much does CARE close this gap, i.e. what is the remaining learning-utilization gap when using CARE?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cWAG

We are very grateful for your valuable comments and questions. We try our best to address your questions as follows.


To W1 (Clarity of the Setup Description):

Thank you for your valuable comment.

1.CARE employs a single-layer linear clustering readout that maps frozen visual features to clustering logits, followed by a standard Softmax function to produce soft clustering assignments, without introducing an additional temperature parameter.

2.CARE utilizes the information obtained by the Inner Learner through an alternating optimization mechanism. Specifically, the Inner Learner is first trained using the soft clustering assignments generated by the current outer clustering readout. Its predicted distributions are then used as supervision to update the parameters of the outer clustering readout through a cross-entropy loss. Through this alternating process, the class-discriminative information encoded in the frozen representations is progressively transferred into clustering assignments.

We have clarified these details in the revised manuscript.


To W2 (Clarity of Ablation and Parameter Sensitivity Results in the Main Paper):

Thank you for your valuable suggestion. We have moved the ablation studies to Section 5.4 of the revised manuscript's main text and briefly summarized the main results of the parameter sensitivity analysis in Section 5.7 of the main text.


To Q1 (Reduction of the Learning–Utilization Gap and Remaining Gap):

Thank you for your insightful question.

We have conducted further analyses using TAC representations, STL-10 and ImageNet-10 are excluded from the statistics because their initial gaps are extremely small. The original and remaining gaps are defined as kkNN oracle accuracy minus TAC and TAC+CARE accuracy, respectively. TAC accuracy is obtained from the original clustering head to measure the overall gap reduction achieved by the complete CARE framework. The results summarized below:

DatasetOriginal Gap (pp)Remaining Gap (pp)Reduction Ratio (%)
CIFAR-101.80.950.0
CIFAR-2026.121.916.1
CIFAR-10015.411.724.0
ImageNet-Dogs6.22.756.5
ImageNet-1K17.616.56.3
DTD16.212.125.3
UCF-10115.110.530.5
Average14.110.929.8

CARE reduces the average Learning–Utilization Gap from 14.1 pp to 10.9 pp, achieving a mean per-dataset gap reduction ratio of 29.8% across 7 datasets. These results demonstrate that CARE can effectively utilize part of the previously underutilized class-related information, although further improvements remain possible. We have incorporated these experiments and analyses into Section 5.3 of the revised manuscript.

Thank you for addressing my points and for uploading the revised manuscript, I have no further questions at this stage.

We are pleased to learn that our responses addressed your concerns. Thank you for your constructive review and for your role in strengthening our manuscript.

Reviewer_Yeo5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper approaches image clustering with the aim of improving the degree to which the information contained in learned image features is utilized in the predictions. It argues that there may be a learning-utilization gap between those features and the final cluster assignments, leaving room for improving the clustering without altering the feature extractor. In addition, it is shown that under certain conditions, a misclassified sample may be corrected using its correctly classified neighbors. Leveraging these findings, the paper introduces CARE, a method that serves as an add-on to existing image clustering approaches to further improve their predictions. Based on the representations learned by the base method, CARE first learns an initial assignment, then it scores each sample's reliability by measuring its agreement with its neighbors. The predictions of the reliable samples are then used to correct the predictions of unreliable ones. Experiments show CARE outperforming its base methods, with a component analysis confirming that all components contribute to the overall result.

Critical strengths
  • The proposed method is intuitive and limits the complexity added on top of the base methods. In addition, CARE delivers a relevant improvement over the respective base methods. The component analysis and the hyperparameter sensitivity study are well executed and demonstrate the value of the components and the robustness of CARE to different parameter choices.
Critical weaknesses
  • The core idea of selecting reliable samples under an existing clustering and using them to further improve the clustering method is not novel, both RUC[1] and DCBoost employ this concept. Especially DCBoost is conceptually close to CARE with its reliance on kNN consistency. These add-on approaches are important baselines for evaluating CARE, more so than the base methods themselves, yet CARE is not compared to them, leaving it unclear if it truly has an advantage over them.
  • The theoretical grounding of CARE appears overstated. While the information theoretical framing of the problem in Theorem 1 is interesting, it is not utilized in the method itself. In addition, Theorem 2 relies on a strong condition that seems more relevant to the method than the exact value for αi\alpha_i. Just from Theorem 2 there appears to be no downside to just setting αi=1\alpha_i=1, this will always result in the correct outcome if the condition on γic\gamma_{ic} is fulfilled. This condition on the other hand is not further elaborated on at all, empirically it may be fulfilled for many samples, but it is not grounded in theory. The relevance of Corollary 1 is entirely unclear, it appears to have no influence on the method design at all and is never referenced after its introduction.

[1] Park, Sungwon, et al. "Improving unsupervised image clustering with robust learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

Questions for authors
  • What is the benefit of setting α\alpha adaptively as αi=1−ri\alpha_i = 1 - r_i, according to Theorem 2 just setting it to 1.0 should also work just as well?
  • Is the accuracy gap between cluster assignments and kNN oracle truly indicative of the learning-utilization gap? Consider a class that is split into two different clusters in the feature space that are both perfectly pure, but far apart from each other. The kNN oracle would achieve 100% accuracy, but clearly the features do not actually contain the knowledge to achieve a 100% accurate clustering, this is only possible given the information that both clusters belong to the same ground truth class, which is information that is external to the features and only contained in the labels.
7 comments
AuthorsOpenReview ↗

Response to Reviewer Yeo5

We are very grateful for your valuable comments and questions. We try our best to address your questions as follows.


To W1 (Comparison with RUC and DCBoost):

Thank you for your valuable suggestion. We would like to clarify that CARE aims to address how class-related information already learned and encoded in frozen visual representations can be more fully translated into final clustering assignments after external knowledge has been incorporated.

1.From a Methodological Perspective: CARE, RUC, and DCBoost utilize reliable samples at different levels. Specifically, RUC selects reliable pseudo-labels from existing clustering results and uses them to retrain the model for improved clustering performance. DCBoost generates pseudo-labels from existing clustering results, identifies high-confidence samples through adaptive kNN neighborhood consistency, and constructs a discriminative loss to fine-tune the model, thereby enhancing intra-class compactness and inter-class separability. In contrast, CARE does not update the learned visual representations. Instead, it improves clustering assignments through Assignment Establishment, which produces stable initial clustering assignments, followed by Selective Assignment Refinement.

2.From an Experimental Perspective: To further evaluate CARE against these plug-in enhancement methods, we have conducted additional comparisons of clustering performance and computational overhead among TAC+RUC, TAC+DCBoost, and TAC+CARE under a unified experimental configuration. The results are summarized below:

DatasetMethodACC (%)Time (s)
CIFAR-100TAC + RUC49.32450.7
TAC + DCBoost58.21277.9
TAC + CARE61.264.6
ImageNet-DogsTAC + RUC84.61297.6
TAC + DCBoost83.32749.3
TAC + CARE85.856.8
UCF-101TAC + RUC20.4646.3
TAC + DCBoost67.8335.0
TAC + CARE72.922.8

(1)CARE achieves the highest ACC on all 3 datasets, outperforming DCBoost by 3.0, 2.5, and 5.1 percentage points, respectively. Moreover, CARE requires approximately 22.8–37.9 times less additional runtime than RUC and 14.7–48.4 times less than DCBoost. These results demonstrate that CARE achieves a favorable trade-off between clustering performance and computational efficiency by more fully utilizing already learned class-related information without re-optimizing the visual representations.

(2)To investigate RUC's performance degradation, we further conduct a post-hoc analysis using ground-truth labels. Although its selected pseudo-labels exhibit high precision, their class coverage is limited on datasets with many categories. For example, only 19 out of 101 ground-truth classes are covered on UCF-101, which may explain its severe performance degradation. In contrast, DCBoost fine-tunes the global feature structure using discriminative losses constructed from reliable samples, avoiding similarly severe degradation in our experiments, although its effectiveness remains dependent on pseudo-label reliability. CARE instead refines cluster assignments without retraining the learned visual representations, reducing reliance on pseudo-label-driven model retraining while achieving higher clustering accuracy.

The corresponding comparisons and analyses have been incorporated into Appendix B.1 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Yeo5

To W2/Q1 (Theoretical Justification and Adaptive Fusion Mechanism):

Thank you for your valuable comments.

1.Regarding the concern about Theorem 1, we clarify that it not only characterizes the Learning–Utilization Gap from an information-theoretic perspective but also guides the design of CARE. Specifically, Theorem 1 motivates CARE to improve the utilization of existing class-related information, which is reflected in its Assignment Establishment, Reliable Assessment, and Knowledge Utilization stages. In the revised manuscript, we further introduce a new theorem that establishes a formal connection between this information-theoretic gap and the accuracy difference between the kNN oracle and the clustering readout. Additional analyses in Section 5.3 quantify how effectively CARE reduces this gap.

2.Regarding the concern about the original Theorem 2.

(1)The condition γic>0\gamma_{ic}>0 is a critical structural prerequisite for guaranteeing error correction, whereas the adaptive fusion coefficient is designed to balance error recovery against the risk of introducing new errors in unsupervised clustering. We acknowledge that when ground-truth labels are known and the condition on γic\gamma_{ic} is satisfied, directly setting αi=1\alpha_i=1 is sufficient to correct an erroneous prediction. However, in practical unsupervised clustering, ground-truth labels are unavailable, making it impossible to determine whether the selected neighborhood satisfies this condition. Consequently, an excessively large fusion coefficient (e.g., αi=1\alpha_i=1) may introduce more errors.

(2)We further conduct experiments to address this concern by comparing fixed fusion with αi=1\alpha_i=1 against adaptive fusion, along with a post-hoc analysis using ground-truth labels. The results are reported below:

DatasetFusionACCNcorrectedN_{\mathrm{corrected}}NintroducedN_{\mathrm{introduced}}ΔN\Delta N
CIFAR-100Fixed61.0650570+80
Adaptive61.2534434+100
ImageNet-DogsFixed82.7118188−70
Adaptive85.810857+51
UCF-101Fixed72.8285213+72
Adaptive72.9257181+76

Although fixed fusion corrects more errors, it also introduces more new errors, particularly on ImageNet-Dogs. Adaptive fusion achieves more net corrections and higher ACC across all three datasets, validating its effectiveness. These results have been added to Appendix F of the revised manuscript.

3.Regarding the concern about Corollary 1, we clarify that it shows that when true same-class neighbors provide positive prediction advantages and the adverse influence of other neighbors is bounded, a sufficiently high local same-class support ratio ρi\rho_i guarantees γic>0\gamma_{ic}>0. Thus, Corollary 1 provides theoretical support for the condition γic>0\gamma_{ic}>0 in the original Theorem 2. Corollary 1 also provides theoretical motivation for CARE's reliable anchor selection and selective neighborhood refinement. We have further clarified the connection between Corollary 1 and the CARE design in the revised manuscript.


To Q2 (Validity of the kNN Oracle Accuracy Gap as an Empirical Diagnostic):

We appreciate your insightful comment. We agree that spatially separated same-class subclusters present a challenge, as unsupervised clustering may fail to recover their class associations using local structure alone.

To assess the practical impact of this scenario, we conduct a post-hoc analysis on frozen TAC representations across all 9 datasets. Specifically, we construct a cosine-based k=20 nearest-neighbor graph and identify same-class subclusters through mutual nearest-neighbor connections using ground-truth labels. Isolated subclusters are defined as secondary subclusters containing at least 21 samples, with at least 95% average neighborhood class purity and significant spatial separation from the main same-class subcluster. We then measure the proportion of Oracle-correct samples belonging to these isolated subclusters, as well as their proportion among samples that are correctly predicted by the Oracle but incorrectly assigned by the clustering method. Under these criteria, no significant isolated subclusters are found in 8 datasets, while ImageNet-10 exhibits only 0.20% among Oracle-correct samples, none of which are misclustered, suggesting that this scenario is rare in our experiments.

Furthermore, the newly introduced Theorem 2 establishes a formal connection between the information-theoretic Learning–Utilization Gap and the kNN Oracle Accuracy Gap, providing theoretical support for using the latter as an empirical diagnostic measure of knowledge utilization. We also provide additional experimental analyses in Section 5.3 to provide empirical support.

Taken together, the theoretical analysis and experimental results suggest that the accuracy gap between clustering assignments and the kNN oracle can serve as a theoretically motivated empirical diagnostic of the Learning–Utilization Gap.

Dear Reviewer Yeo5,

We sincerely thank you for your thorough review of our paper and for providing constructive feedback that has significantly contributed to its improvement. Your insights have been invaluable in helping us refine our work.

As the discussion period is nearing its end, we would like to ensure that we have addressed your concerns satisfactorily. If there are any remaining questions or additional suggestions, please let us know. We would be happy to address any further concerns to improve our work.

Best regards,

The authors.

Reviewer_owVKOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a learning–utilization gap in external-knowledge-assisted image clustering: category information encoded in representations may not be reflected in final assignments. It proposes CARE, which establishes assignments on frozen features, selects reliable anchors through neighborhood consistency, and adaptively refines unreliable predictions. Experiments cover nine datasets, two clustering pipelines, and three external knowledge sources.

Critical strengths
  1. CARE consistently improves TAC and SATC across diverse datasets without updating the knowledge-enhanced representations. On UCF101, ACC increases from 68.3% to 72.9% for TAC and from 70.7% to 74.2% for SATC.

  2. The appendix includes full factorial component ablations, alternative readout architectures, anchor-quality analysis, and parameter sensitivity across five seeds.

Critical weaknesses
  1. The theoretical gap concerns information in soft predictions, whereas the empirical gap measures oracle–assignment accuracy differences. The connection between these quantities is not established.

  2. Component ablations do not establish CARE’s advantage over standard neighborhood refinement using identical frozen features and initial assignments.

  3. Figure 2 is embedded as a raster image. A vector version would preserve the clarity of its text and equations when enlarged.

CARE shows consistent empirical benefits and includes substantial ablation analysis. However, the mismatch between theory and measurement, together with missing matched refinement baselines, leaves the central conceptual and methodological contributions insufficiently established.

Questions for authors
  1. Can the authors establish a formal connection between the oracle–readout accuracy difference and I(Y;Z∣P)I(Y;Z\mid P), accounting for the distinction between soft predictions and hard assignments?

  2. Using ground truth only for analysis, how often do CARE’s actual anchors and blending coefficients satisfy Theorem 2’s correction condition?

  3. Does CARE outperform matched neighborhood-refinement baselines across repeated runs, particularly where the incremental gains are small?

5 comments
AuthorsOpenReview ↗

Response to Reviewer owVK

We are very grateful for your valuable comments and questions. We try our best to address your questions as follows.


To W1/Q1 (Theoretical Connection Between the Learning–Utilization Gap and Accuracy Gap):

Thank you for your valuable suggestion. In the revised manuscript, we introduce a new Theorem 2 to establish a formal connection between the Learning–Utilization Gap and the accuracy difference between the kNN oracle and the clustering readout, based on Bayes-optimal classification accuracy. Specifically, we derive GACC≤I(Y;Z∣P)/2+εdec\mathcal{G}_{\mathrm{ACC}} \le \sqrt{I(Y;Z\mid P)/2}+\varepsilon_{\mathrm{dec}}, where GACC\mathcal{G}_{\mathrm{ACC}} denotes the accuracy gap between the kNN oracle and the clustering readout, I(Y;Z∣P)I(Y;Z\mid P) quantifies the Learning–Utilization Gap, and εdec\varepsilon_{\mathrm{dec}} characterizes the accuracy discrepancy between soft predictions and hard clustering decisions.

This inequality provides theoretical justification for using the accuracy gap as an empirical diagnostic measure of the Learning–Utilization Gap, while accounting for the distinction between soft predictions and hard clustering decisions through the conditional entropy relationship induced by H=arg⁡max⁡cPcH=\arg\max_c P_c. The complete proof has been included in the Appendix C.2.


To W2/Q3 (Comparison with Neighborhood Refinement Baseline Across Multiple Seeds):

Thank you for your insightful comment. We have conducted additional comparisons across multiple random seeds. We would like to clarify that CARE improves clustering assignments on frozen visual representations that have already incorporated external knowledge through both Assignment Establishment and Selective Assignment Refinement. Establishing stable initial assignments is therefore an integral component of CARE. Therefore, to fairly compare the complete CARE framework with standard neighborhood refinement, we use the same TAC representations and implement the baseline by generating original soft assignments from the clustering head used during external knowledge learning, then averaging the soft predictions of each sample's k=20 nearest neighbors to obtain refined assignments. The mean and standard deviation of ACC (%) over 5 random seeds are reported below:

DatasetNeighborhood Refinement BaselineCARE
CIFAR-1090.46±1.0692.64±0.05
ImageNet-1099.37±0.0199.40±0.00
CIFAR-10058.86±0.7961.44±0.80
ImageNet-Dogs83.64±1.1585.84±0.09
UCF-10169.00±0.5172.62±0.79

CARE achieves higher mean ACC than the baseline on 5 representative datasets. Furthermore, we have included a seed-wise paired comparison across different random seeds in the revised manuscript, as shown in Appendix B.3. These results demonstrate that CARE consistently outperforms the neighborhood refinement baseline across repeated runs using the same frozen TAC representations and paired random seeds.


To W3 (Clarity of Figure 2):

We appreciate your constructive suggestion. We have improved the clarity of Figure 2 in the revised manuscript.


To Q2 (Empirical Verification of the Error-Correction Conditions):

We appreciate your valuable question. We have conducted an additional post-hoc analysis in which ground-truth labels are used exclusively for evaluation. For samples with incorrect initial predictions that satisfy the neighborhood structural condition γic>0,  ∀c≠yi\gamma_{ic}>0,\;\forall c\neq y_i, we measure the proportion for which the fusion coefficients actually adopted by CARE satisfy the correction condition in Theorem 2: αi>max⁡c≠yi[δic]+[δic]++γic\alpha_i > \max_{c\neq y_i} \frac{[\delta_{ic}]_+}{[\delta_{ic}]_+ + \gamma_{ic}}.

The results show that 92.5%, 96.4%, and 93.6% of these samples satisfy the condition on CIFAR-100, ImageNet-Dogs, and UCF-101, respectively. These findings provide empirical support for the applicability of Theorem 2 to the actual behavior of CARE.

Dear Reviewer owVK,

We sincerely thank you for your thorough review of our paper and for providing constructive feedback that has significantly contributed to its improvement. Your insights have been invaluable in helping us refine our work.

As the discussion period is nearing its end, we would like to ensure that we have addressed your concerns satisfactorily. If there are any remaining questions or additional suggestions, please let us know. We would be happy to address any further concerns to improve our work.

Best regards,

The authors.

Thank you for the rebuttal, which addressed several questions. However, I agree with other reviewers that the theoretical proof of CARE appears overstated, and should be improved.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Key Improvements and New Additions

We sincerely thank all reviewers for their constructive comments. We are grateful that reviewers recognized CARE's novel focus on improving external knowledge utilization, its consistent gains across 9 datasets and 3 external knowledge sources, and the effectiveness and robustness of its components. We have tried our best to address the reviews’ comments and questions. Based on your suggestions, we have significantly improved the manuscript's quality, specifically by strengthening theoretical foundations, expanding baseline comparisons, deepening mechanism validation, and clarifying methodological details. The key improvements and new additions are summarized as follows:


Key Improvements and Clarifications

  1. In-Depth Theoretical Analysis
  • We established a formal connection between theoretical and empirical measures of the Learning–Utilization Gap.
  • We clarified how the theoretical results motivate and guide CARE's methodological design and implementation details, and emphasized the theoretical and methodological connection of adaptive fusion, strengthening the connection between theory and methodology.
  1. Extended Experimental Evaluation and Analysis

(1) Expanded Baseline Comparisons

  • We compared CARE with plug-in clustering enhancement methods (RUC and DCBoost), further demonstrating CARE's performance advantages.
  • We compared CARE with a standard neighborhood refinement baseline, further validating its effectiveness and stability.

(2) Deeper Mechanism Validation

  • We quantified the reduction and remaining gap after applying CARE, validating its effectiveness in reducing the Learning–Utilization Gap.
  • We compared fixed fusion (αi=1\alpha_i=1) with adaptive fusion (αi=1−ri\alpha_i=1-r_i), demonstrating the effectiveness of adaptive fusion.
  • We evaluated the Learning–Utilization Gap using a supervised MLP oracle, confirming that empirical gaps persist under an alternative diagnostic.
  1. Methodological Clarifications
  • We clarified the default clustering readout architecture and how the clustering model incorporates information from the Inner Learner.
  • We improved the clarity of Figure 2.

New Additions in the Revised Version

  1. We have introduced a new Theorem 2 in Section 3 to establish a formal connection between the Learning–Utilization Gap and the empirical kNN oracle accuracy gap. Asked by Reviewer owVK.

  2. We have added experiments in Section 5.3 to validate CARE's effectiveness in reducing the Learning–Utilization Gap. Asked by Reviewer cWAG.

  3. We have added Appendix B.1, which compares clustering performance and computational efficiency with plug-in clustering enhancement methods (RUC and DCBoost). Asked by Reviewer Yeo5.

  4. We have added Appendix B.3 to compare CARE with the standard neighborhood refinement baseline across different random seeds. Asked by Reviewer owVK.

  5. We have added Appendix F to compare fixed fusion (αi=1\alpha_i=1) with adaptive fusion (αi=1−ri\alpha_i=1-r_i), and provide theoretical justification of why larger fusion coefficients may be counterproductive, the connection between the fusion coefficient αi\alpha_i in Theorem 3 and Eq. (17), and the theoretical motivation for adaptive fusion. Asked by Reviewer Yeo5.

  6. We have added Appendix K to evaluate the Learning–Utilization Gap using a supervised MLP oracle. Asked by Reviewer Yeo5.


Please see our detailed responses below each review, and all the improvements in the revised paper are highlighted. We appreciate your feedback and welcome any additional questions.

AuthorsOpenReview ↗

Summary of Contributions for Area Chair

Dear Area Chair,

We sincerely appreciate the ICLR organizing committee's efforts to ensure a fair and rigorous review process. To facilitate a clear and efficient understanding of our work, we summarize the motivation, contributions, and experimental findings of our manuscript as follows.


Motivation Highlight

Existing external-knowledge-assisted image clustering methods primarily focus on learning useful external knowledge into visual representations, while overlooking whether the learned knowledge is effectively utilized in final clustering assignments. Consequently, category-related information may already be encoded in the representations but remain underutilized in clustering predictions, leading to a Learning–Utilization Gap. This motivates us to more fully exploit the category-related information already encoded in frozen visual representations to reduce the Learning–Utilization Gap and thereby improve clustering performance.


Contribution Highlight

  1. We theoretically characterize the Learning–Utilization Gap to formulate CARE’s information-theoretic objective and establish the kNN oracle accuracy gap as a theoretically motivated empirical diagnostic for assessing the potential to better utilize learned external knowledge through CARE’s design. We further establish sufficient conditions for neighborhood-based error correction, while demonstrating that stronger class-consistent structural support improves the correction guarantee.

  2. We propose CARE, a theory-guided framework that consists of three core components:

  • Assignment Establishment: Establishes stable initial cluster assignments through iterative optimization to better utilize category-related information encoded in frozen visual representations.
  • Reliability Assessment: Identifies reliable anchors based on local neighborhood consistency, motivated by the theoretical importance of class-consistent structural support.
  • Knowledge Utilization: Adaptively refines unreliable assignments using predictions from reliable neighbors, guided by Theorem 3’s correction condition and Eq. (37)’s explanation of the risks of excessive fusion.
  1. We demonstrate CARE's effectiveness and generality across 9 benchmark datasets and 3 external knowledge sources.

Experimental (Performance) Support

  1. Comparative Experiments (Tables 1, 2, 7, 8 and Figure 6): We compare CARE with a wide range of classical and state-of-the-art clustering approaches, including external-knowledge-assisted clustering methods (TAC, SATC, etc.), plug-in clustering enhancement methods (RUC and DCBoost), and a standard neighborhood refinement baseline. CARE achieves the best performance.

  2. Mechanism Validation of Knowledge Utilization:

  • Existence of the Learning–Utilization Gap (Figure 3 and Table 13): We evaluate the Learning–Utilization Gap using both kNN and supervised MLP oracles, demonstrating the existence of the Learning–Utilization Gap.
  • Reduction of the Learning–Utilization Gap (Table 3): We evaluate the kNN oracle accuracy gap before and after applying CARE, demonstrating CARE's effectiveness in reducing the Learning–Utilization Gap.
  • Structural Support and Error Recovery (Figure 4): We analyze the relationship between class-consistent structural support and error recovery, demonstrating that stronger structural support facilitates the correction of erroneous assignments.
  • Reliability of Selected Anchors (Figure 7): We evaluate the prediction accuracy of selected anchors, demonstrating that CARE's reliability assessment effectively identifies reliable samples for knowledge utilization.
  1. Component Ablation (Table 4): We conduct a comprehensive component ablation study, demonstrating the contributions of CARE's components.

  2. Generality Experiments:

  • External Knowledge Sources (Table 5): We validate CARE's generality across 3 different external knowledge sources.
  • Clustering Readouts (Table 11): We validate CARE's generality across 3 different clustering readouts.
  • Multiple Auxiliary Modalities: We extend CARE to settings with 2 auxiliary modalities (see Appendix H for details), demonstrating its effectiveness in multimodal knowledge utilization.
  1. Effectiveness Analysis:
  • Assignment Establishment (Table 9): We analyze the effects of different optimization iterations, demonstrating the effectiveness and efficiency.
  • Adaptive Fusion (Table 10): We compare fixed and adaptive fusion strategies, demonstrating that adaptive fusion achieves more net corrections and higher clustering accuracy, while emphasizing the theoretical and methodological connection of adaptive fusion and providing theoretical justification.
  1. Visualization, Parameter Sensitivity, and Time Analysis (Figures 5, 8 and Table 12): We visualize CARE's progressive error recovery, analyze its parameter sensitivity, and evaluate its computational cost, demonstrating its effectiveness, robustness, and efficiency.
AuthorsOpenReview ↗

Summary of Rebuttal Discussion for Area Chair

We sincerely appreciate the Area Chair's efforts and thank all reviewers for their time and constructive feedback. The following summary provides a concise overview of the discussion during the rebuttal phase.


Key concerns from Reviewer owVK and our responses:

  1. Request a formal connection between the Learning–Utilization Gap and the empirical kNN oracle accuracy gap.
  • We derived an accuracy-gap bound based on Bayes-optimal accuracy and conditional mutual information, demonstrating that the kNN oracle accuracy gap can serve as a theoretically motivated empirical diagnostic of the Learning–Utilization Gap.
  1. Question whether CARE consistently outperforms standard neighborhood refinement across multiple random seeds.
  • We conducted paired comparisons across 5 random seeds using the same frozen visual representations. CARE consistently outperforms the baseline across all 5 random seeds, demonstrating its effectiveness and stability.
  1. Hope to know the proportion of samples for which CARE satisfies Theorem 2's correction condition in practice.
  • We conducted a post-hoc analysis and found that 92.5%, 96.4%, and 93.6% of the samples satisfying Theorem 2's correction condition on CIFAR-100, ImageNet-Dogs, and UCF-101, respectively.
  1. Suggest improving the clarity of Figure 2.
  • We improved the clarity.
  1. Claim that CARE’s theoretical contributions are overstated without identifying specific shortcomings.
  • Although we requested further clarification, we received no response. Despite this, we clarified the scope of our theoretical analysis and its connection to CARE's methodological design, supported by empirical mechanism validation. We also emphasized CARE’s contributions in problem formulation, methodological design, and comprehensive experimental evaluation.

Key concerns from Reviewer Yeo5 and our responses:

  1. Ask for comparisons between CARE and plug-in clustering enhancement methods (RUC and DCBoost).
  • We evaluated RUC and DCBoost, demonstrating CARE's consistent advantages in clustering performance and computational efficiency.
  1. Request stronger connections between the theoretical results and the methodological design.
  • We clarified that Theorem 1 formulates CARE’s information-theoretic objective. Furthermore, we introduced a new theorem establishing a formal connection between theoretical and empirical measures of the Learning–Utilization Gap for assessing the potential to better utilize learned external knowledge through CARE’s design, and provided additional mechanism validation experiments to demonstrate CARE's effectiveness in reducing this gap.

  • We clarified that the original Theorem 2 establishes that an initially misclustered sample with positive neighborhood support can be corrected when αi\alpha_i exceeds the theoretical threshold. Such samples represent an important form of knowledge underutilization that motivates CARE. Furthermore, we introduced Eq. (37), which, together with the original Theorem 2, theoretically motivates controlling fusion strength to balance error recovery against newly introduced errors.

  • We clarified that Corollary 1 shows that a sufficiently high local same-class support ratio guarantees the positive structural support condition in the original Theorem 2, while also motivating CARE's reliable anchor selection and selective neighborhood refinement.

  1. Question whether the kNN oracle accuracy gap can reflect the Learning–Utilization Gap, using isolated same-class subclusters as a counterexample.
  • We conducted a post-hoc analysis showing that isolated same-class subclusters are rare across all 9 datasets in our experiments. Furthermore, the newly introduced Theorem 2 and additional mechanism validation experiments jointly provide theoretical and empirical support for using the kNN oracle accuracy gap as a theoretically motivated empirical diagnostic of the Learning–Utilization Gap.
  1. Hope for a stronger theoretical and methodological connection for adaptive fusion.
  • We provided additional theoretical analyses explaining why larger fusion coefficients may be counterproductive and clarified the connection between Theorem 3 and Eq. (17), providing theoretical justification for the adaptive fusion strategy.
  1. Request validation of the Learning–Utilization Gap using a supervised MLP oracle.
  • We introduced a supervised MLP oracle evaluation, confirming that empirical gaps persist under an alternative diagnostic.

Key concerns from Reviewer cWAG and our responses:

  1. Ask about the reduction and remaining gap after applying CARE.
  • We quantified the reduction and remaining gap after applying CARE, validating its effectiveness.
  1. Hope for clarification of the setup description and improvements to the paper's presentation.
  • We clarified the description and improved the paper's presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.