acceptodds
Under review as a conference paper at ICLR 2027

Calibrated Category Discovery: Knitting Calibration from the Known into the Novel

Abstract

Generalized Category Discovery (GCD) aims to recognize old classes while discovering novel ones in unlabeled data, yet the practical value of such discoveries hinges on whether the model's confidence can be trusted. Existing GCD methods optimize clustering accuracy alone, leaving their predictions on novel classes markedly miscalibrated. We argue that this deficiency cannot be resolved by conventional post-hoc calibration. First, miscalibration in GCD is inherently cluster-level, whereas conventional methods apply a single sample-level mapping to all clusters. Second, novel classes provide no labels for calibrating confidence. To address this, we formulate Calibrated Category Discovery (CalCD) and introduce a cluster-conditional expected calibration error (cluECE) that assesses calibration within each predicted cluster, preventing the deviations of different clusters from cancelling as they do in the standard expected calibration error. We further propose Known-to-Novel calIbration Transfer (KNIT), grounded in the premise that the trustworthiness of a cluster may follow a class-agnostic pattern. KNIT characterizes cluster reliability through the model's self-assessment and an independent semantic check by a vision-language model, estimates this pattern on labeled old-class clusters, and transfers it to novel-class clusters, correcting confidence without altering predictions. Extensive experiments comprising 220 comparisons across 4 GCD methods, 11 datasets spanning generic, fine-grained and multi-domain images, and 5 calibration baselines demonstrate that KNIT significantly reduces cluECE for both novel and old classes.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Calibration, Category Discovery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_M4YEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argue that "what kind of cluster deserves trust" may be a class-agnostic pattern in category discovery. The paper proposes cluECE that measures the model's condifence on each discovered group, rather than on average. Then it proposed the method KNIT, which adjust per-cluster confidence. The adjustment is through an offset that is determined by whether a cluster deserves trust from both external and internal views.

Critical strengths

The research question is of importance and the writing of the paper is easy to understand.

Critical weaknesses
  1. The sample size is limited. In the paper, around 10 to 40 images are in each test group for the fine-grained and office-home data. The accuracy obtained in those data may be affected by sampling bias. How do you trust or testify your claim under such smaller-size setup?

  2. The concept of cluECE and the top-label ECE by Gupta & Ramdas (2022) look quite similar if I remember correclty. The authors should state the difference. However, the paper is only cited in the appendix. The authors should position their paper against Gupta & Ramdas (2022) in the introduction. In that case, contribution (1) should be rephrased.

  3. Related to the previous comments, the authors should also include top-label binning on the known clusters in their experiments.

  4. In Table 4, the conclusion that the trust pattern carries over from known to novel classes does not persuade me. The paper reports that the two methods (the proposed method and the method fitting with novel-class lables) has hiigh rank correlation (0.87). This agreement may be due to the fact that both methods use the same inputs on the same groups. The author should compare each with the per-class correction fitted with novel-vlass labels, and compare with a baseline (weights fitted on a random permuted labels).

5.The independent vision-language model check is actually dependent. It uses the same CLIP model and the same word list as SpectralGCD. CLIP is weak in handling medical images (where this paper is motivated by), where the paper did not include any medical image examples. The authors should try a different "independent" model and try on a medical image dataset that CLIP does not perform well.

  1. We need a hold-out set for the data used for calibration. Right now, the paper use data in the test set for calibration.

  2. Each model in the experiments appears to be trained once?The paper mentioned "the final model is used". The bootstrap intervals only reports test-sample variation. What about the training variation?

Questions for authors

Please address the 7 comments in the Weaknesses.

Reviewer_sR1DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces CalCD (Calibrated Category Discovery) and cluster-conditional expected calibration error (cluECE). Generic GCD uses clustering accuracy and does not calibrate on novel classes. This paper aims to define a new error (cluECE) to consider this cluster-level miscalibration of GCD and a new method named KNIT that tries to learn class-agnostic patterns to determine the confidence of a model.

Critical strengths

1- Making models to calibrate their confidence and prevent them from being overconfident in their assignment seems interesting.

2- Transferring class-agnostic knowledge from known to novel categories to estimate the cluster confidence seems reasonable.

Critical weaknesses

1- The writing of the paper is hard to follow; there are a lot of forward references, and many parts of the paper are, in fact, delegated to the Appendix. This makes it hard to understand the paper in one read, end to end.

2- While the paper has introduced a new cluster ECE, this is not a major contribution, and it is a very simple reweighting.

3- The paper then introduces an overly complicated method to decrease this specific measure. While the ablations in Table 2 suggest that the conf and marg components are nearly unnecessary, the paper fails to provide a convincing experimental claim for these components.

4- The paper is supposed to determine a new problem called CalCD; however, the definition is very general, and it is not clear what the difference is from typical calibrated classification.

5- Tables C2 and C3 in the appendix make the adoption of KNIT quite unconvincing; these tables show no improvement on accuracy while also sometimes a drop in it. While Table 1 showed that KNIT improves cluECE. What this shows is probably both KNIT and cluECE are not necessary good.

Minor: Figure 2 is quite small and unreadable. Sometimes there are more than 3-5 equation references in different parts of the paper; this could be written in a much more coherent way. It makes it very hard for the reader to consider many different small equations in different parts of the paper.

Questions for authors

1- One thing that didn't convince me in reading the paper is what the benefit of CalCD is. There are some arguments about its medical use in the introduction, but does it lend confidence to the typical GCD?

2- In Table 2, why marg and conf components has been used?

Reviewer_nj56OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines Calibrated Category Discovery: adjust confidence without changing predictions, using labels only from known classes. cluECE computes ECE inside each predicted cluster. KNIT adds one logit-space offset per cluster on top of temperature scaling. The offset is a linear function of relative confidence, a CLIP margin, a CLIP purity, and cluster size.

Critical strengths
  1. The task is set at the right place. Usual post-hoc calibration fits one sample-level map and needs labels for every class. GCD has neither: the error differs by cluster and cancels in ECE, and novel clusters have no labels. CalCD asks for cluster-conditional calibration from known-class labels only. cluECE scores each cluster before averaging, so errors of opposite sign cannot cancel.

  2. KNIT is built for those two constraints. One offset per cluster can raise some clusters and lower others; a global temperature cannot. The offset is read from label-free descriptors, fit on known clusters, and applied to novel ones. Level constants keep each group at its own temperature-scaling confidence, so only the relative pattern is transferred. Clipping leaves the predicted cluster unchanged.

Critical weaknesses
  1. The results do not support the transfer claim at the strength of the abstract. The headline change is 15.53 to 13.44. Four of 44 combinations are exactly temperature scaling, because CIFAR-10 has fewer than 10 clusters per type. A per-cluster correction that uses novel labels still reaches 9.64, so most of the error remains. Table 4 treats a weaker label-using fit as a reference, but KNIT beats it (transfer efficiency 1.16 overall, 1.38 on domain shift). That column is not an upper bound and does not show that the transferred pattern matches fitting on novel labels.

  2. Descriptors and level constants are computed on the unlabeled training set, not on held-out data. The shared coefficients are pooled from known-class test labels of 18 reported combinations. The external check is the same CLIP ViT-B/16 used to initialize every backbone. On long-tailed CIFAR-100, those level constants freeze the temperature-scaling bias: at imbalance 100, novel cluECE is 24.5 versus 23.6, and the uncalibrated model wins the "All" column at imbalances 50 and 100. The supervision-based known/novel split also disagrees with the Hungarian split used by cluECE, especially in the long-tailed case.

Questions for authors
  1. Report novel cluECE only on combinations excluded from the pooled coefficient fit (SSR2-GCD, Flowers-102, Office-Home). Confirm that novel test labels never enter Eq. 16.
  2. What is fit in Table 4's "novel-class labels" column, and why is it worse than KNIT? How does it differ from the per-cluster correction that reaches 9.64?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.