- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argue that "what kind of cluster deserves trust" may be a class-agnostic pattern in category discovery. The paper proposes cluECE that measures the model's condifence on each discovered group, rather than on average. Then it proposed the method KNIT, which adjust per-cluster confidence. The adjustment is through an offset that is determined by whether a cluster deserves trust from both external and internal views.
Critical strengths
The research question is of importance and the writing of the paper is easy to understand.
Critical weaknesses
-
The sample size is limited. In the paper, around 10 to 40 images are in each test group for the fine-grained and office-home data. The accuracy obtained in those data may be affected by sampling bias. How do you trust or testify your claim under such smaller-size setup?
-
The concept of cluECE and the top-label ECE by Gupta & Ramdas (2022) look quite similar if I remember correclty. The authors should state the difference. However, the paper is only cited in the appendix. The authors should position their paper against Gupta & Ramdas (2022) in the introduction. In that case, contribution (1) should be rephrased.
-
Related to the previous comments, the authors should also include top-label binning on the known clusters in their experiments.
-
In Table 4, the conclusion that the trust pattern carries over from known to novel classes does not persuade me. The paper reports that the two methods (the proposed method and the method fitting with novel-class lables) has hiigh rank correlation (0.87). This agreement may be due to the fact that both methods use the same inputs on the same groups. The author should compare each with the per-class correction fitted with novel-vlass labels, and compare with a baseline (weights fitted on a random permuted labels).
5.The independent vision-language model check is actually dependent. It uses the same CLIP model and the same word list as SpectralGCD. CLIP is weak in handling medical images (where this paper is motivated by), where the paper did not include any medical image examples. The authors should try a different "independent" model and try on a medical image dataset that CLIP does not perform well.
-
We need a hold-out set for the data used for calibration. Right now, the paper use data in the test set for calibration.
-
Each model in the experiments appears to be trained once?The paper mentioned "the final model is used". The bootstrap intervals only reports test-sample variation. What about the training variation?
Questions for authors
Please address the 7 comments in the Weaknesses.