- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates generalized category discovery (GCD) with pretrained vision-language models, arguing that novel categories exhibit different levels of familiarity in the CLIP representation space, making uniform semantic guidance potentially unreliable. The authors propose FAME, a Familiarity-Aware Mixture of Energies framework consisting of three components: (1) Counterfactual Familiarity Estimation (CFE), which measures the stability and discriminability of CLIP-space assignments under perturbations; (2) Semantic Intervention Risk (SIR), which estimates potential semantic guidance errors through visual-semantic conflicts and assignment ambiguity; and (3) Risk-Aware Energy Routing (RAER), which adaptively combines semantic and visual-structural evidence based on cluster familiarity and risk.
The method is evaluated on six GCD benchmarks under both known-K and unknown-K settings. The reported results show improvements over existing methods, particularly on fine-grained datasets. Additional experiments, including routing ablations, familiarity and risk analyses, initialization robustness, and comparisons across different foundation encoders, provide empirical support for the proposed framework.
Critical strengths
- The paper addresses a meaningful issue in CLIP-based category discovery. The observation that novel categories can differ in how reliably they are represented in CLIP space is interesting. Rather than applying the same semantic guidance to all clusters, FAME considers both semantic familiarity and the risk of misleading assignments. This provides a reasonable motivation for adaptive semantic routing, particularly for fine-grained categories where pretrained semantic information may be unreliable.
- The experimental results provide good support for the proposed approach. FAME improves over recent methods on six benchmarks, with particularly clear gains under the unknown-K setting. The comparisons with simpler routing strategies in Table 3 help demonstrate the benefit of combining familiarity and risk estimation. The additional experiments on initialization stability and different foundation encoders also strengthen the results, suggesting that the improvements are not limited to a particular initialization or CLIP model.
Critical weaknesses
- The definition of familiarity is not entirely convincing. In Eqs. (3)–(8), familiarity is estimated from assignment stability under perturbations and the margin between CLIP-space prototypes. However, these quantities mainly measure clustering confidence and stability rather than how familiar a category is to the pretrained model. Although the authors provide text-based validation in Appendix C, it is still unclear whether CFam can distinguish genuinely familiar categories from confidently clustered but semantically unfamiliar ones. This distinction matters because familiarity is the central motivation of the paper.
- The formulation of semantic intervention risk is questionable. Eq. (10) treats disagreement between visual and CLIP assignments as a sign of harmful semantic guidance. However, disagreement does not necessarily mean that CLIP is incorrect. In some cases, the visual branch may produce an incorrect assignment while CLIP provides useful semantic information. Suppressing semantic guidance in such cases could reinforce errors in the visual representation. The paper does not clearly establish whether the proposed risk score can distinguish harmful semantic intervention from useful correction.
- The routing mechanism depends heavily on the quality of the initial clusters. In Eq. (17), the routing coefficient is determined by familiarity and risk scores associated with the current cluster assignment. Since these scores are themselves computed from evolving assignments and prototypes, incorrect early assignments may affect subsequent routing decisions. Although Table 10 examines noisy initialization, it only considers perturbed assignments and does not establish how the method behaves when initial clusters contain substantial class mixing or systematic errors. This raises concerns about the reliability of the iterative refinement process.
- The novelty of the routing formulation appears limited. Eqs. (17)–(19) combine familiarity and risk through a simple multiplicative coefficient and use it to interpolate between semantic and visual energies. Both familiarity and risk are constructed from existing measures such as assignment consistency, prototype margins, prediction entropy, and cross-view disagreement. Although Table 3 compares several simpler gates, the paper does not sufficiently explain why this particular formulation is preferable to other ways of combining these reliability signals. The improvements are encouraging, but the methodological contribution seems largely based on combining established confidence measures rather than introducing a fundamentally new learning mechanism.
Questions for authors
- How can the authors distinguish semantic familiarity from general clustering confidence? Eqs. (3)–(8) mainly rely on assignment stability and prototype margins, which may also be high for semantically unfamiliar but visually compact clusters. Although Appendix C reports correlations with CLIP text embeddings, can the authors provide evidence that CFam identifies semantic familiarity beyond these generic confidence measures?
- In Eq. (10), disagreement between visual and CLIP assignments is treated as semantic intervention risk. However, such disagreement may also occur when CLIP provides the correct assignment and the visual branch is wrong. How does FAME distinguish harmful semantic guidance from useful semantic correction? Could the authors analyze the accuracy of each branch on samples with high conflict scores?
- How sensitive is FAME to errors in the initial cluster assignments? Since the familiarity and risk scores depend on the current assignments and prototypes, incorrect clusters may influence subsequent routing decisions. Table 10 considers noisy initialization, but does the method remain effective under systematic cluster errors or substantial mixing of different categories within the same cluster?
- What justifies the multiplicative routing rule in Eq. (17)? The formulation assumes that familiarity and intervention risk should be combined as CFam(k)(1 − Risk(k)), but the paper does not establish why this interaction is preferable to additive weighting or a learned routing function. Given the comparisons with simpler gates in Table 3, could the authors clarify whether the multiplicative interaction itself provides a measurable benefit beyond using the two reliability scores separately?
4 comments
Q2. How does FAME distinguish harmful semantic guidance from useful semantic correction?
Our existing results already provide consistent evidence supporting the risk score. shows that high-risk clusters are more susceptible to negative transfer from fixed semantic fusion, while FAME improves their accuracy. further shows that risk scores computed before convergence predict smaller or negative subsequent gains from semantic guidance. Together, these results support the effectiveness of risk estimation at the cluster level.
While these results support the effectiveness of risk estimation at the cluster level, we appreciate the reviewer’s suggestion to examine a complementary question: whether FAME preserves useful semantic corrections when the visual branch is wrong. To address this point, we added a targeted branch-correctness analysis on high-conflict samples, measuring both the reduction of harmful guidance and the preservation of useful semantic corrections.
Specifically, we select clusters in the highest third of the conflict score on CUB and analyze samples where the two branches disagree. Fixed fusion, CFam-only routing, and FAME use the same saved representations and prototypes, allowing us to isolate the immediate effect of the fusion rule.
| Method | Accuracy (%) |
|---|---|
| Semantic branch | |
| Visual branch | |
| Fixed fusion | |
| CFam-only routing | |
| FAME |
We further separate samples according to which branch is correct:
| Conflict case | Share (%) | Mean Risk | Mean semantic weight | CFam-only accuracy (%) | FAME accuracy (%) |
|---|---|---|---|---|---|
| Semantic correct / visual wrong | 0.41 | 0.53 | |||
| Visual correct / semantic wrong | 0.71 | 0.12 | |||
| Both wrong | 0.78 | 0.09 |
These results show the trade-off directly. FAME retains 84.4% of useful semantic corrections while achieving 88.1% accuracy when the visual branch is correct and the semantic branch is wrong. Compared with CFam-only routing, risk suppression decreases useful-correction accuracy from 87.3% to 84.4%, but increases accuracy in potentially harmful-guidance cases from 61.6% to 88.1%. Thus, the additional analysis evaluates both the benefit and the cost of risk suppression. It supports the intended role of the score as a statistical estimate of intervention risk, without assuming that disagreement always indicates an incorrect semantic prediction.
Q3. How robust is FAME to systematic errors in the initial clusters?
Our existing results already support FAME’s robustness to initialization errors. In , noisy initialization reduces FAME’s CUB accuracy from 86.8% to 85.9%, compared with a decrease from 84.2% to 80.7% for fixed fusion. The stabilization ablations further support the contributions of warm-up, routing ramp-up, and EMA smoothing.
While these results provide evidence under assignment perturbations, we appreciate the reviewer’s suggestion to examine more structured errors. We therefore extended the analysis with two additional initialization settings:
- Pairwise mixing: exchange 10%, 30%, or 50% of samples between nearby novel-category clusters selected using warm-up features.
- Merge-and-split: merge nearby clusters and fill the freed slots by splitting other clusters, keeping the total category count unchanged. Severity denotes the proportion of novel slots participating in merges.
| Corruption | Severity | Fixed fusion: New | w/o RR+EMA: New | FAME: All | FAME: Old | FAME: New | FAME NMI: Init → E20 → E200 |
|---|---|---|---|---|---|---|---|
| Pairwise mixing | 10% | 0.64 → 0.76 → 0.84 | |||||
| Pairwise mixing | 30% | 0.58 → 0.72 → 0.81 | |||||
| Pairwise mixing | 50% | 0.54 → 0.65 → 0.76 | |||||
| Merge-and-split | 10% | 0.66 → 0.75 → 0.84 | |||||
| Merge-and-split | 20% | 0.64 → 0.70 → 0.81 | |||||
| Merge-and-split | 30% | 0.61 → 0.64 → 0.76 |
The comparison directly examines recovery from structured errors. Under 30% pairwise mixing, FAME achieves 87.9% New accuracy, compared with 79.6% for fixed fusion and 81.3% without ramp-up and score smoothing. Its novel-sample NMI increases from 0.58 to 0.81 during refinement, showing improvement from the corrupted initialization.
The severe-corruption settings also expose the limits of recovery. With 30% of novel slots involved in merging, FAME’s New accuracy decreases to 81.6%, below its uncorrupted value of 90.3%. Thus, the results support improved resilience and partial recovery, without implying recovery from arbitrary incorrect initializations.
Q4. What justifies the multiplicative routing rule, and does the interaction itself provide a measurable benefit?
① Why multiplication rather than additive weighting or learned routing?
The multiplicative rule in expresses the requirement that semantic guidance should be both familiar and low-risk. Let and . The product satisfies and , allowing either weak familiarity or high intervention risk to limit semantic influence.
For example, when and , multiplication assigns a semantic weight of 0.09, whereas arithmetic averaging assigns 0.50. characterizes the resulting bounds on semantic influence. These properties motivate the design, but do not establish its empirical superiority over alternative combinations. We appreciate the reviewer’s suggestion to examine this choice directly.
We therefore compare multiplication with arithmetic averaging, a tuned weighted sum, minimum gating, and a learned two-input MLP. All rules receive the same familiarity and safety statistics and share the reference construction, warm-up initialization, objectives, and training budget. The weighted-sum coefficient is selected using held-out old-class validation data. The MLP takes only and , adds 33 trainable parameters, and is trained with the same GCD objectives.
| Routing rule | All | Old | New |
|---|---|---|---|
| CFam-only: | |||
| Safety-only: | |||
| Arithmetic mean: | |||
| Tuned weighted sum: | |||
| Minimum: | |||
| Learned gate: | |||
| FAME: |
② Does the multiplicative interaction provide a measurable benefit beyond using the scores separately?
Yes. already provides the relevant single-score comparisons:
- CFam-only routing uses familiarity alone.
- Risk-only routing uses safety alone.
- Full FAME combines both through multiplication.
On CUB, full FAME achieves 86.8% All accuracy, compared with 85.4% for CFam-only and 85.1% for Risk-only, yielding improvements of 1.4 and 1.7 percentage points. New accuracy similarly increases from 88.7% and 88.2% to 90.4%, respectively. Full FAME also outperforms both single-score variants in All and New accuracy across all six datasets.
These results support a measurable benefit from jointly using the two scores through the multiplicative gate over either score alone. The additional comparisons discussed above () address the more specific question of whether multiplication is preferable to other functions that also use both scores.
Q1. How do we distinguish semantic familiarity from generic clustering confidence?
① Clarifying the meaning of familiarity.
We thank the reviewer for highlighting this distinction. We use “familiarity” to mean how reliably a discovered cluster is represented in the pretrained semantic reference space. CFam is a composite proxy for this reliability, built from perturbation stability and prototype separation. It does not establish whether a category appeared in CLIP’s pretraining data, and a high score does not guarantee a correct semantic interpretation.
For example, images of an unfamiliar bird species may form a compact visual cluster, while the semantic reference still confuses that species with several alternatives. Visual compactness alone cannot resolve this distinction. Conversely, a confidently incorrect semantic cluster may also receive a high CFam score. We therefore assess CFam against semantic recognizability and the actual benefit of semantic guidance, rather than treating stability itself as proof of familiarity.
② Existing evidence and additional confidence-controlled validation.
Appendix C, Tables 4–5 links CFam to text-based semantic recognizability, while Appendix D.1, Tables 6–7 links it to the benefit of semantic guidance. We agree, however, that these correlations alone do not establish whether CFam provides information beyond generic clustering confidence. We therefore supplement them with two controlled analyses on CUB.
Table R1. Confidence-controlled comparison on CUB.
Table R2. Additional control for prototype margin on CUB.
Together, these analyses support CFam as a proxy for semantic-reference reliability beyond the tested generic confidence measures.