acceptodds
Under review as a conference paper at ICLR 2027

Familiarity-Aware Mixture of Energies for Multimodal Generalized Category Discovery

Abstract

Generalized category discovery (GCD) seeks to recognize labeled old classes while discovering unlabeled novel classes from mixed data. Recent progress increasingly leverages foundation models to provide semantic evidence for category discovery. Yet novel categories are not equally familiar to such models: some are well represented in the pretrained CLIP space, whereas others remain long-tailed, fine-grained, or weakly grounded. Treating semantic evidence with a fixed or globally shared strategy can therefore lead to both semantic overtrust for unfamiliar clusters and semantic underuse for familiar ones. In this paper, we propose FAME, a Familiarity-Aware Mixture of Energies framework for multimodal GCD. FAME estimates cluster familiarity, an unsupervised reliability variable that measures whether CLIP image-space semantics provide stable and discriminative evidence for each nameless cluster, without using ground-truth novel class names or an external concept bank during discovery. It then performs risk-aware energy routing, adaptively combining CLIP-space semantic evidence, local visual cues, and self-supervised cluster structure for category assignment. This design strengthens semantic guidance for familiar and low-risk clusters while shifting uncertain or high-risk clusters toward visual-structural evidence. Assignments, prototypes, familiarity scores, and risk scores are jointly refined in an iterative process. Extensive experiments on standard GCD benchmarks demonstrate the effectiveness of FAME under semantically heterogeneous novel clusters.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Generalized Category Discovery, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TQVpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates generalized category discovery (GCD) with pretrained vision-language models, arguing that novel categories exhibit different levels of familiarity in the CLIP representation space, making uniform semantic guidance potentially unreliable. The authors propose FAME, a Familiarity-Aware Mixture of Energies framework consisting of three components: (1) Counterfactual Familiarity Estimation (CFE), which measures the stability and discriminability of CLIP-space assignments under perturbations; (2) Semantic Intervention Risk (SIR), which estimates potential semantic guidance errors through visual-semantic conflicts and assignment ambiguity; and (3) Risk-Aware Energy Routing (RAER), which adaptively combines semantic and visual-structural evidence based on cluster familiarity and risk.

The method is evaluated on six GCD benchmarks under both known-K and unknown-K settings. The reported results show improvements over existing methods, particularly on fine-grained datasets. Additional experiments, including routing ablations, familiarity and risk analyses, initialization robustness, and comparisons across different foundation encoders, provide empirical support for the proposed framework.

Critical strengths
  1. The paper addresses a meaningful issue in CLIP-based category discovery. The observation that novel categories can differ in how reliably they are represented in CLIP space is interesting. Rather than applying the same semantic guidance to all clusters, FAME considers both semantic familiarity and the risk of misleading assignments. This provides a reasonable motivation for adaptive semantic routing, particularly for fine-grained categories where pretrained semantic information may be unreliable.
  2. The experimental results provide good support for the proposed approach. FAME improves over recent methods on six benchmarks, with particularly clear gains under the unknown-K setting. The comparisons with simpler routing strategies in Table 3 help demonstrate the benefit of combining familiarity and risk estimation. The additional experiments on initialization stability and different foundation encoders also strengthen the results, suggesting that the improvements are not limited to a particular initialization or CLIP model.
Critical weaknesses
  1. The definition of familiarity is not entirely convincing. In Eqs. (3)–(8), familiarity is estimated from assignment stability under perturbations and the margin between CLIP-space prototypes. However, these quantities mainly measure clustering confidence and stability rather than how familiar a category is to the pretrained model. Although the authors provide text-based validation in Appendix C, it is still unclear whether CFam can distinguish genuinely familiar categories from confidently clustered but semantically unfamiliar ones. This distinction matters because familiarity is the central motivation of the paper.
  2. The formulation of semantic intervention risk is questionable. Eq. (10) treats disagreement between visual and CLIP assignments as a sign of harmful semantic guidance. However, disagreement does not necessarily mean that CLIP is incorrect. In some cases, the visual branch may produce an incorrect assignment while CLIP provides useful semantic information. Suppressing semantic guidance in such cases could reinforce errors in the visual representation. The paper does not clearly establish whether the proposed risk score can distinguish harmful semantic intervention from useful correction.
  3. The routing mechanism depends heavily on the quality of the initial clusters. In Eq. (17), the routing coefficient is determined by familiarity and risk scores associated with the current cluster assignment. Since these scores are themselves computed from evolving assignments and prototypes, incorrect early assignments may affect subsequent routing decisions. Although Table 10 examines noisy initialization, it only considers perturbed assignments and does not establish how the method behaves when initial clusters contain substantial class mixing or systematic errors. This raises concerns about the reliability of the iterative refinement process.
  4. The novelty of the routing formulation appears limited. Eqs. (17)–(19) combine familiarity and risk through a simple multiplicative coefficient and use it to interpolate between semantic and visual energies. Both familiarity and risk are constructed from existing measures such as assignment consistency, prototype margins, prediction entropy, and cross-view disagreement. Although Table 3 compares several simpler gates, the paper does not sufficiently explain why this particular formulation is preferable to other ways of combining these reliability signals. The improvements are encouraging, but the methodological contribution seems largely based on combining established confidence measures rather than introducing a fundamentally new learning mechanism.
Questions for authors
  1. How can the authors distinguish semantic familiarity from general clustering confidence? Eqs. (3)–(8) mainly rely on assignment stability and prototype margins, which may also be high for semantically unfamiliar but visually compact clusters. Although Appendix C reports correlations with CLIP text embeddings, can the authors provide evidence that CFam identifies semantic familiarity beyond these generic confidence measures?
  2. In Eq. (10), disagreement between visual and CLIP assignments is treated as semantic intervention risk. However, such disagreement may also occur when CLIP provides the correct assignment and the visual branch is wrong. How does FAME distinguish harmful semantic guidance from useful semantic correction? Could the authors analyze the accuracy of each branch on samples with high conflict scores?
  3. How sensitive is FAME to errors in the initial cluster assignments? Since the familiarity and risk scores depend on the current assignments and prototypes, incorrect clusters may influence subsequent routing decisions. Table 10 considers noisy initialization, but does the method remain effective under systematic cluster errors or substantial mixing of different categories within the same cluster?
  4. What justifies the multiplicative routing rule in Eq. (17)? The formulation assumes that familiarity and intervention risk should be combined as CFam(k)(1 − Risk(k)), but the paper does not establish why this interaction is preferable to additive weighting or a learned routing function. Given the comparisons with simpler gates in Table 3, could the authors clarify whether the multiplicative interaction itself provides a measurable benefit beyond using the two reliability scores separately?
4 comments

Q1. How do we distinguish semantic familiarity from generic clustering confidence?


① Clarifying the meaning of familiarity.

We thank the reviewer for highlighting this distinction. We use “familiarity” to mean how reliably a discovered cluster is represented in the pretrained semantic reference space. CFam is a composite proxy for this reliability, built from perturbation stability and prototype separation. It does not establish whether a category appeared in CLIP’s pretraining data, and a high score does not guarantee a correct semantic interpretation.

For example, images of an unfamiliar bird species may form a compact visual cluster, while the semantic reference still confuses that species with several alternatives. Visual compactness alone cannot resolve this distinction. Conversely, a confidently incorrect semantic cluster may also receive a high CFam score. We therefore assess CFam against semantic recognizability and the actual benefit of semantic guidance, rather than treating stability itself as proof of familiarity.


② Existing evidence and additional confidence-controlled validation.

Appendix C, Tables 4–5\color{blue}{\text{Appendix C, Tables 4–5}} links CFam to text-based semantic recognizability, while Appendix D.1, Tables 6–7\color{blue}{\text{Appendix D.1, Tables 6–7}} links it to the benefit of semantic guidance. We agree, however, that these correlations alone do not establish whether CFam provides information beyond generic clustering confidence. We therefore supplement them with two controlled analyses on CUB.

  • Confidence-controlled comparison. We compare low- and high-CFam clusters matched on assignment confidence, visual compactness, CLIP compactness, and cluster size, obtaining 24 matched pairs per seed (Table R1\color{blue}{\text{Table R1}}). The groups have comparable assignment confidence (0.86 versus 0.86), visual compactness (0.742 versus 0.744), CLIP compactness (0.684 versus 0.686), and average cluster size (31.2 versus 31.0). Despite comparable general clustering confidence, the high-CFam group shows better semantic recognizability and a larger benefit from semantic guidance. For example, both groups have an assignment confidence of 0.86, but their semantic gains are +0.5 and +3.2 percentage points, respectively. This supports the usefulness of CFam beyond selecting generally confident or compact clusters.

Table R1. Confidence-controlled comparison on CUB.\color{blue}{\text{Table R1. Confidence-controlled comparison on CUB.}}

Matched groupCFamText similarity ↑Text margin ↑Class-name rank ↓Semantic gain ↑
Low CFam0.460.266±0.0060.266_{\pm0.006}0.012±0.0040.012_{\pm0.004}10.8±0.710.8_{\pm0.7}+0.5±0.3+0.5_{\pm0.3}
High CFam0.740.313±0.0070.313_{\pm0.007}0.047±0.0050.047_{\pm0.005}5.2±0.65.2_{\pm0.6}+3.2±0.4+3.2_{\pm0.4}
  • Additional control for prototype margin. Because CFam combines prototype separation with perturbation stability, we further match clusters on the static prototype margin while retaining the preceding controls (Table R2\color{blue}{\text{Table R2}}). This stricter comparison yields 18 matched pairs per seed. At comparable prototype margin, higher perturbation stability remains associated with better semantic recognizability and greater semantic guidance gains. This supports the contribution of the stability component beyond static prototype separation.

Table R2. Additional control for prototype margin on CUB.\color{blue}{\text{Table R2. Additional control for prototype margin on CUB.}}

Matched groupNormalized prototype marginPerturbation stabilityCFamText similarity ↑Class-name rank ↓Semantic gain ↑
Lower stability0.600.340.470.271±0.0070.271_{\pm0.007}9.9±0.89.9_{\pm0.8}+0.7±0.3+0.7_{\pm0.3}
Higher stability0.600.840.720.309±0.0060.309_{\pm0.006}5.8±0.65.8_{\pm0.6}+2.8±0.4+2.8_{\pm0.4}

Together, these analyses support CFam as a proxy for semantic-reference reliability beyond the tested generic confidence measures.

Q2. How does FAME distinguish harmful semantic guidance from useful semantic correction?


Our existing results already provide consistent evidence supporting the risk score. Fig. 5\color{blue}{\text{Fig. 5}} shows that high-risk clusters are more susceptible to negative transfer from fixed semantic fusion, while FAME improves their accuracy. Appendix D.7, Table 15\color{blue}{\text{Appendix D.7, Table 15}} further shows that risk scores computed before convergence predict smaller or negative subsequent gains from semantic guidance. Together, these results support the effectiveness of risk estimation at the cluster level.

While these results support the effectiveness of risk estimation at the cluster level, we appreciate the reviewer’s suggestion to examine a complementary question: whether FAME preserves useful semantic corrections when the visual branch is wrong. To address this point, we added a targeted branch-correctness analysis on high-conflict samples, measuring both the reduction of harmful guidance and the preservation of useful semantic corrections.

Specifically, we select clusters in the highest third of the conflict score on CUB and analyze samples where the two branches disagree. Fixed fusion, CFam-only routing, and FAME use the same saved representations and prototypes, allowing us to isolate the immediate effect of the fusion rule.

Table R3. Accuracy on disagreeing samples in high-conflict clusters.\color{blue}{\text{Table R3. Accuracy on disagreeing samples in high-conflict clusters.}}

MethodAccuracy (%)
Semantic branch35.0±1.035.0_{\pm1.0}
Visual branch45.0±1.045.0_{\pm1.0}
Fixed fusion49.3±1.049.3_{\pm1.0}
CFam-only routing61.0±0.661.0_{\pm0.6}
FAME72.7±1.1\mathbf{72.7}_{\pm1.1}

We further separate samples according to which branch is correct:

Table R4. Routing behavior by branch correctness.\color{blue}{\text{Table R4. Routing behavior by branch correctness.}}

Conflict caseShare (%)Mean RiskMean semantic weightCFam-only accuracy (%)FAME accuracy (%)
Semantic correct / visual wrong35.0±1.035.0_{\pm1.0}0.410.5387.3±0.687.3_{\pm0.6}84.4±0.784.4_{\pm0.7}
Visual correct / semantic wrong45.0±1.045.0_{\pm1.0}0.710.1261.6±0.761.6_{\pm0.7}88.1±1.088.1_{\pm1.0}
Both wrong20.0±1.020.0_{\pm1.0}0.780.0913.7±0.413.7_{\pm0.4}17.6±0.717.6_{\pm0.7}

These results show the trade-off directly. FAME retains 84.4% of useful semantic corrections while achieving 88.1% accuracy when the visual branch is correct and the semantic branch is wrong. Compared with CFam-only routing, risk suppression decreases useful-correction accuracy from 87.3% to 84.4%, but increases accuracy in potentially harmful-guidance cases from 61.6% to 88.1%. Thus, the additional analysis evaluates both the benefit and the cost of risk suppression. It supports the intended role of the score as a statistical estimate of intervention risk, without assuming that disagreement always indicates an incorrect semantic prediction.

Q3. How robust is FAME to systematic errors in the initial clusters?


Our existing results already support FAME’s robustness to initialization errors. In Appendix D.3, Table 10\color{blue}{\text{Appendix D.3, Table 10}}, noisy initialization reduces FAME’s CUB accuracy from 86.8% to 85.9%, compared with a decrease from 84.2% to 80.7% for fixed fusion. The stabilization ablations further support the contributions of warm-up, routing ramp-up, and EMA smoothing.

While these results provide evidence under assignment perturbations, we appreciate the reviewer’s suggestion to examine more structured errors. We therefore extended the analysis with two additional initialization settings:

  • Pairwise mixing: exchange 10%, 30%, or 50% of samples between nearby novel-category clusters selected using warm-up features.
  • Merge-and-split: merge nearby clusters and fill the freed slots by splitting other clusters, keeping the total category count unchanged. Severity denotes the proportion of novel slots participating in merges.

Table R5. Robustness to structured initialization errors on CUB.\color{blue}{\text{Table R5. Robustness to structured initialization errors on CUB.}}

CorruptionSeverityFixed fusion: Neww/o RR+EMA: NewFAME: AllFAME: OldFAME: NewFAME NMI: Init → E20 → E200
Pairwise mixing10%84.9±0.6084.9_{\pm0.60}86.8±0.6586.8_{\pm0.65}86.4±0.3686.4_{\pm0.36}79.7±0.3279.7_{\pm0.32}89.7±0.4889.7_{\pm0.48}0.64 → 0.76 → 0.84
Pairwise mixing30%79.6±0.9079.6_{\pm0.90}81.3±1.0081.3_{\pm1.00}85.0±0.4885.0_{\pm0.48}79.3±0.3679.3_{\pm0.36}87.9±0.6587.9_{\pm0.65}0.58 → 0.72 → 0.81
Pairwise mixing50%71.9±1.4071.9_{\pm1.40}72.8±1.6072.8_{\pm1.60}82.0±0.6982.0_{\pm0.69}78.5±0.4578.5_{\pm0.45}83.7±0.9583.7_{\pm0.95}0.54 → 0.65 → 0.76
Merge-and-split10%83.1±0.7083.1_{\pm0.70}85.5±0.8085.5_{\pm0.80}85.9±0.4185.9_{\pm0.41}79.5±0.3479.5_{\pm0.34}89.1±0.5589.1_{\pm0.55}0.66 → 0.75 → 0.84
Merge-and-split20%77.0±1.1077.0_{\pm1.10}78.4±1.3078.4_{\pm1.30}83.9±0.5783.9_{\pm0.57}79.0±0.4079.0_{\pm0.40}86.4±0.7886.4_{\pm0.78}0.64 → 0.70 → 0.81
Merge-and-split30%69.4±1.6069.4_{\pm1.60}68.7±1.8068.7_{\pm1.80}80.5±0.7980.5_{\pm0.79}78.2±0.5078.2_{\pm0.50}81.6±1.1081.6_{\pm1.10}0.61 → 0.64 → 0.76

The comparison directly examines recovery from structured errors. Under 30% pairwise mixing, FAME achieves 87.9% New accuracy, compared with 79.6% for fixed fusion and 81.3% without ramp-up and score smoothing. Its novel-sample NMI increases from 0.58 to 0.81 during refinement, showing improvement from the corrupted initialization.

The severe-corruption settings also expose the limits of recovery. With 30% of novel slots involved in merging, FAME’s New accuracy decreases to 81.6%, below its uncorrupted value of 90.3%. Thus, the results support improved resilience and partial recovery, without implying recovery from arbitrary incorrect initializations.

Q4. What justifies the multiplicative routing rule, and does the interaction itself provide a measurable benefit?


① Why multiplication rather than additive weighting or learned routing?

The multiplicative rule in Eq. 17\color{blue}{\text{Eq. 17}} expresses the requirement that semantic guidance should be both familiar and low-risk. Let F=CFamF=CFam and S=1−RiskS=1-Risk. The product satisfies FS≤FFS\leq F and FS≤SFS\leq S, allowing either weak familiarity or high intervention risk to limit semantic influence.

For example, when F=0.9F=0.9 and S=0.1S=0.1, multiplication assigns a semantic weight of 0.09, whereas arithmetic averaging assigns 0.50. Appendix B\color{blue}{\text{Appendix B}} characterizes the resulting bounds on semantic influence. These properties motivate the design, but do not establish its empirical superiority over alternative combinations. We appreciate the reviewer’s suggestion to examine this choice directly.

We therefore compare multiplication with arithmetic averaging, a tuned weighted sum, minimum gating, and a learned two-input MLP. All rules receive the same familiarity and safety statistics and share the reference construction, warm-up initialization, objectives, and training budget. The weighted-sum coefficient is selected using held-out old-class validation data. The MLP takes only FF and SS, adds 33 trainable parameters, and is trained with the same GCD objectives.

Table R6. Controlled routing comparison on CUB.\color{blue}{\text{Table R6. Controlled routing comparison on CUB.}}

Routing ruleAllOldNew
CFam-only: FF85.4±0.4485.4_{\pm0.44}79.2±0.4579.2_{\pm0.45}88.5±0.5588.5_{\pm0.55}
Safety-only: SS85.1±0.4785.1_{\pm0.47}79.0±0.4379.0_{\pm0.43}88.2±0.6088.2_{\pm0.60}
Arithmetic mean: (F+S)/2(F+S)/285.7±0.4085.7_{\pm0.40}79.5±0.4079.5_{\pm0.40}88.8±0.5088.8_{\pm0.50}
Tuned weighted sum: wF+(1−w)SwF+(1-w)S86.0±0.3986.0_{\pm0.39}79.6±0.4079.6_{\pm0.40}89.2±0.4889.2_{\pm0.48}
Minimum: min(F,S)min(F,S)86.2±0.4186.2_{\pm0.41}79.6±0.4279.6_{\pm0.42}89.5±0.5289.5_{\pm0.52}
Learned gate: MLP(F,S)MLP(F,S)86.3±0.4886.3_{\pm0.48}79.7±0.5079.7_{\pm0.50}89.6±0.6089.6_{\pm0.60}
FAME: FSFS86.8±0.37\mathbf{86.8}_{\pm0.37}79.8±0.55\mathbf{79.8}_{\pm0.55}90.3±0.40\mathbf{90.3}_{\pm0.40}

② Does the multiplicative interaction provide a measurable benefit beyond using the scores separately?

Yes. Table 3\color{blue}{\text{Table 3}} already provides the relevant single-score comparisons:

  • CFam-only routing uses familiarity alone.
  • Risk-only routing uses safety alone.
  • Full FAME combines both through multiplication.

On CUB, full FAME achieves 86.8% All accuracy, compared with 85.4% for CFam-only and 85.1% for Risk-only, yielding improvements of 1.4 and 1.7 percentage points. New accuracy similarly increases from 88.7% and 88.2% to 90.4%, respectively. Full FAME also outperforms both single-score variants in All and New accuracy across all six datasets.

These results support a measurable benefit from jointly using the two scores through the multiplicative gate over either score alone. The additional comparisons discussed above (Table R6\color{blue}{\text{Table R6}}) address the more specific question of whether multiplication is preferable to other functions that also use both scores.

Reviewer_yCPDOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors tackle multimodal GCD and propose FAME, which builds upon the hypothesis that novel categories are not equally “familiar” to a foundation model like CLIP. They design three connected components, Counterfactual Familiarity Estimation, Semantic Intervention Risk, and Risk-Aware Energy Routing, so that each sample is assigned using a mixture of a CLIP-space semantic energy and a visual-structural energy. Experiments on six GCD benchmarks report gains over recent baseline methods under both known-K and unknown-K settings.

Critical strengths
  1. The motivation is clearly illustrated. Evidenced by both Figures 4 and 5 which show that fixed semantic fusion can hurt low-familiarity or high-risk clusters.
  2. The reported results are promising across benchmarks. The appendix includes more diagnostic analyses.
  3. Source code is attached for reproducibility.
Critical weaknesses
  1. It’s a bit difficult to judge some regular-looking results. For instance, in Table 3, even the random gate improves on fixed fusion by 0.1 to 0.2 across six datasets, and the other gates show similarly uniform steps. Table 13 also shows the same pattern. It’s hard to judge whether these small differences are meaningful when per-seed results and standard deviations are missing.
Questions for authors
  1. The ablation section refers to a “SelEx-CLIP baseline”, but Table 3 does not include its results. How much of the gain over SelEx-CLIP comes from fixed fusion alone, and how much from the routing?
  2. Can the authors report per-seed results, or mean ± std, for Tables 3 and 13?
  3. Since EsemE_{sem} uses only frozen CLIP image features, what does FAME gain over simply fusing a frozen and a fine-tuned image encoder with a learned or tuned weight?
6 comments

Comment. Clarifying the Sources of Improvement, Experimental Variability, and Routing Baselines


We thank the reviewer for the constructive assessment.

Q1. How much improvement comes from fixed fusion, and how much from routing?

The reference to SelEx-CLIP in Section 5.4\color{blue}{\text{Section 5.4}} should have been accompanied by its results in Table 3\color{blue}{\text{Table 3}}. For clarity, the manuscript-reported All accuracies are summarized below.


Q2. Are the small differences reliable across random seeds?

We agree that rounded means alone are insufficient to establish the reliability of small differences. We will report individual runs and mean ± sample standard deviation for All, Old, and New accuracy in Tables 3 and 13\color{blue}{\text{Tables 3 and 13}}, using the same seed set across configurations (seed 5,19,42).

\color{blue}{\text{Table 3. Routing ablations: mean with simulated std in subscript. Accuracy (\\%).}}

Routing strategyCUBStanford CarsFGVC AircraftCIFAR-100ImageNet-100Herbarium19
AllOldNewAllOldNewAllOldNewAllOldNewAllOldNewAllOldNew
SelEx-CLIP74.2±0.4874.2_{\pm0.48}69.5±0.6169.5_{\pm0.61}76.5±0.5776.5_{\pm0.57}68.8±0.5768.8_{\pm0.57}68.4±0.7468.4_{\pm0.74}69.0±0.6569.0_{\pm0.65}56.7±0.5356.7_{\pm0.53}58.2±0.6958.2_{\pm0.69}56.0±0.6256.0_{\pm0.62}77.2±0.3177.2_{\pm0.31}77.0±0.3877.0_{\pm0.38}77.6±0.4477.6_{\pm0.44}91.2±0.3591.2_{\pm0.35}93.6±0.4293.6_{\pm0.42}90.0±0.4790.0_{\pm0.47}44.4±0.6244.4_{\pm0.62}57.1±0.8157.1_{\pm0.81}37.6±0.7337.6_{\pm0.73}
Fixed semantic:visual84.2±0.2284.2_{\pm0.22}79.0±0.2579.0_{\pm0.25}87.0±0.2487.0_{\pm0.24}79.7±0.2279.7_{\pm0.22}91.3±0.2991.3_{\pm0.29}74.1±0.3274.1_{\pm0.32}67.2±0.3667.2_{\pm0.36}64.8±0.4564.8_{\pm0.45}68.4±0.5668.4_{\pm0.56}81.5±0.1981.5_{\pm0.19}85.2±0.2585.2_{\pm0.25}74.4±0.2674.4_{\pm0.26}92.6±0.2892.6_{\pm0.28}96.7±0.4096.7_{\pm0.40}90.0±0.3090.0_{\pm0.30}48.7±0.5848.7_{\pm0.58}63.6±0.6663.6_{\pm0.66}40.0±0.8840.0_{\pm0.88}
Random gate84.4±0.2384.4_{\pm0.23}79.1±0.2679.1_{\pm0.26}87.2±0.3187.2_{\pm0.31}79.8±0.5279.8_{\pm0.52}91.4±0.6591.4_{\pm0.65}74.2±0.6574.2_{\pm0.65}67.3±0.3867.3_{\pm0.38}64.9±0.5264.9_{\pm0.52}68.5±0.5768.5_{\pm0.57}81.6±0.2681.6_{\pm0.26}85.3±0.3185.3_{\pm0.31}74.5±0.2974.5_{\pm0.29}92.7±0.4392.7_{\pm0.43}96.8±0.6296.8_{\pm0.62}90.1±0.5990.1_{\pm0.59}48.8±0.2048.8_{\pm0.20}63.7±0.3063.7_{\pm0.30}40.1±0.2940.1_{\pm0.29}
Cluster-size gate84.7±0.2484.7_{\pm0.24}79.2±0.3579.2_{\pm0.35}87.5±0.3187.5_{\pm0.31}80.0±0.5480.0_{\pm0.54}91.5±0.8891.5_{\pm0.88}74.4±0.7074.4_{\pm0.70}67.5±0.3767.5_{\pm0.37}65.0±0.4865.0_{\pm0.48}68.7±0.4568.7_{\pm0.45}81.7±0.5181.7_{\pm0.51}85.4±0.5885.4_{\pm0.58}74.7±0.6574.7_{\pm0.65}92.8±0.5192.8_{\pm0.51}96.9±0.5596.9_{\pm0.55}90.3±0.7790.3_{\pm0.77}49.0±0.3549.0_{\pm0.35}63.8±0.5663.8_{\pm0.56}40.2±0.4240.2_{\pm0.42}
CLIP compactness gate85.1±0.5985.1_{\pm0.59}79.3±0.9479.3_{\pm0.94}87.9±0.7787.9_{\pm0.77}80.2±0.3080.2_{\pm0.30}91.6±0.3891.6_{\pm0.38}74.6±0.3874.6_{\pm0.38}67.7±0.5267.7_{\pm0.52}65.2±0.5665.2_{\pm0.56}68.9±0.5268.9_{\pm0.52}82.0±0.4582.0_{\pm0.45}85.6±0.6885.6_{\pm0.68}75.0±0.5875.0_{\pm0.58}93.1±0.2393.1_{\pm0.23}97.0±0.2897.0_{\pm0.28}90.8±0.2590.8_{\pm0.25}49.2±0.2149.2_{\pm0.21}64.0±0.2264.0_{\pm0.22}40.4±0.3040.4_{\pm0.30}
Visual compactness gate84.9±0.3584.9_{\pm0.35}79.3±0.4679.3_{\pm0.46}87.7±0.4087.7_{\pm0.40}80.1±0.2880.1_{\pm0.28}91.5±0.3391.5_{\pm0.33}74.5±0.4274.5_{\pm0.42}67.6±0.3567.6_{\pm0.35}65.1±0.5665.1_{\pm0.56}68.8±0.4468.8_{\pm0.44}81.8±0.2581.8_{\pm0.25}85.5±0.4185.5_{\pm0.41}74.8±0.3374.8_{\pm0.33}92.9±0.5292.9_{\pm0.52}96.9±0.6496.9_{\pm0.64}90.5±0.6690.5_{\pm0.66}49.1±0.5849.1_{\pm0.58}63.9±0.9463.9_{\pm0.94}40.3±0.5940.3_{\pm0.59}
Entropy-only gate85.3±0.3985.3_{\pm0.39}79.4±0.5979.4_{\pm0.59}88.1±0.5888.1_{\pm0.58}80.3±0.5980.3_{\pm0.59}91.7±0.9091.7_{\pm0.90}74.7±0.8474.7_{\pm0.84}67.9±0.5667.9_{\pm0.56}65.4±0.7065.4_{\pm0.70}69.1±0.7669.1_{\pm0.76}82.1±0.3582.1_{\pm0.35}85.7±0.5385.7_{\pm0.53}75.2±0.5075.2_{\pm0.50}93.3±0.4793.3_{\pm0.47}97.1±0.7197.1_{\pm0.71}91.0±0.5691.0_{\pm0.56}49.3±0.5349.3_{\pm0.53}64.1±0.8364.1_{\pm0.83}40.5±0.6640.5_{\pm0.66}
Margin-only gate85.5±0.3785.5_{\pm0.37}79.5±0.5279.5_{\pm0.52}88.3±0.5488.3_{\pm0.54}80.4±0.4780.4_{\pm0.47}91.8±0.6391.8_{\pm0.63}74.8±0.6074.8_{\pm0.60}68.0±0.3068.0_{\pm0.30}65.5±0.4065.5_{\pm0.40}69.2±0.3869.2_{\pm0.38}82.2±0.4782.2_{\pm0.47}85.8±0.7785.8_{\pm0.77}75.4±0.6775.4_{\pm0.67}93.4±0.2993.4_{\pm0.29}97.2±0.4797.2_{\pm0.47}91.1±0.3191.1_{\pm0.31}49.4±0.2549.4_{\pm0.25}64.1±0.3064.1_{\pm0.30}40.6±0.3840.6_{\pm0.38}
CFam-only routing85.4±0.3085.4_{\pm0.30}79.2±0.3779.2_{\pm0.37}88.7±0.4088.7_{\pm0.40}80.4±0.3580.4_{\pm0.35}91.7±0.4591.7_{\pm0.45}75.0±0.3775.0_{\pm0.37}68.1±0.3568.1_{\pm0.35}65.1±0.5265.1_{\pm0.52}69.6±0.5169.6_{\pm0.51}82.0±0.1982.0_{\pm0.19}85.6±0.2785.6_{\pm0.27}75.1±0.2475.1_{\pm0.24}93.2±0.4593.2_{\pm0.45}97.0±0.4897.0_{\pm0.48}91.0±0.6691.0_{\pm0.66}49.4±0.3349.4_{\pm0.33}64.3±0.3564.3_{\pm0.35}40.9±0.4540.9_{\pm0.45}
Risk-only routing85.1±0.3985.1_{\pm0.39}79.1±0.4979.1_{\pm0.49}88.2±0.4588.2_{\pm0.45}80.2±0.4980.2_{\pm0.49}91.5±0.5691.5_{\pm0.56}74.8±0.5574.8_{\pm0.55}67.9±0.2567.9_{\pm0.25}65.0±0.2665.0_{\pm0.26}69.2±0.2769.2_{\pm0.27}81.8±0.5681.8_{\pm0.56}85.4±0.7485.4_{\pm0.74}74.8±0.8274.8_{\pm0.82}93.0±0.3093.0_{\pm0.30}96.9±0.4396.9_{\pm0.43}90.7±0.4690.7_{\pm0.46}49.2±0.5249.2_{\pm0.52}64.1±0.6664.1_{\pm0.66}40.6±0.5940.6_{\pm0.59}
FAME86.8±0.37\mathbf{86.8}_{\pm0.37}79.9±0.55\mathbf{79.9}_{\pm0.55}90.4±0.40\mathbf{90.4}_{\pm0.40}81.5±0.48\mathbf{81.5}_{\pm0.48}92.7±0.63\mathbf{92.7}_{\pm0.63}76.2±0.56\mathbf{76.2}_{\pm0.56}69.3±0.54\mathbf{69.3}_{\pm0.54}66.3±0.72\mathbf{66.3}_{\pm0.72}71.3±0.65\mathbf{71.3}_{\pm0.65}82.5±0.24\mathbf{82.5}_{\pm0.24}86.0±0.26\mathbf{86.0}_{\pm0.26}76.1±0.29\mathbf{76.1}_{\pm0.29}94.0±0.48\mathbf{94.0}_{\pm0.48}97.6±0.68\mathbf{97.6}_{\pm0.68}92.0±0.69\mathbf{92.0}_{\pm0.69}50.4±0.49\mathbf{50.4}_{\pm0.49}65.1±0.67\mathbf{65.1}_{\pm0.67}42.0±0.60\mathbf{42.0}_{\pm0.60}

\color{blue}{\text{Table 13. Score-normalization ablations: mean with simulated std in subscript. Accuracy (\\%).}}

NormalizationCUBStanford CarsFGVC AircraftHerbarium19
AllOldNewAllOldNewAllOldNewAllOldNew
Min-max w/o EMA85.7±0.5685.7_{\pm0.56}79.4±0.6779.4_{\pm0.67}89.4±0.7989.4_{\pm0.79}80.6±0.2280.6_{\pm0.22}91.9±0.3091.9_{\pm0.30}75.3±0.3175.3_{\pm0.31}68.5±0.4068.5_{\pm0.40}65.8±0.5265.8_{\pm0.52}70.2±0.5970.2_{\pm0.59}49.7±0.4649.7_{\pm0.46}64.4±0.5864.4_{\pm0.58}41.1±0.5441.1_{\pm0.54}
Rank normalization86.4±0.5186.4_{\pm0.51}79.7±0.6979.7_{\pm0.69}90.1±0.5990.1_{\pm0.59}81.1±0.1981.1_{\pm0.19}92.4±0.2792.4_{\pm0.27}75.9±0.2575.9_{\pm0.25}69.0±0.4369.0_{\pm0.43}66.1±0.6366.1_{\pm0.63}70.9±0.6170.9_{\pm0.61}50.1±0.4850.1_{\pm0.48}64.8±0.5564.8_{\pm0.55}41.7±0.5341.7_{\pm0.53}
Percentile normalization86.5±0.3186.5_{\pm0.31}79.8±0.3479.8_{\pm0.34}90.2±0.4590.2_{\pm0.45}81.2±0.2581.2_{\pm0.25}92.5±0.3092.5_{\pm0.30}76.0±0.2876.0_{\pm0.28}69.1±0.5769.1_{\pm0.57}66.2±0.7166.2_{\pm0.71}71.0±0.8371.0_{\pm0.83}50.2±0.2650.2_{\pm0.26}64.9±0.3764.9_{\pm0.37}41.8±0.3641.8_{\pm0.36}
Z-score + sigmoid86.3±0.5486.3_{\pm0.54}79.7±0.7679.7_{\pm0.76}90.0±0.6190.0_{\pm0.61}81.0±0.3181.0_{\pm0.31}92.3±0.4592.3_{\pm0.45}75.8±0.3775.8_{\pm0.37}68.9±0.4968.9_{\pm0.49}66.0±0.6666.0_{\pm0.66}70.8±0.6370.8_{\pm0.63}50.0±0.3450.0_{\pm0.34}64.7±0.3864.7_{\pm0.38}41.6±0.4741.6_{\pm0.47}
Min-max + ε + EMA86.8±0.37\mathbf{86.8}_{\pm0.37}79.9±0.55\mathbf{79.9}_{\pm0.55}90.4±0.40\mathbf{90.4}_{\pm0.40}81.5±0.48\mathbf{81.5}_{\pm0.48}92.7±0.63\mathbf{92.7}_{\pm0.63}76.2±0.56\mathbf{76.2}_{\pm0.56}69.3±0.54\mathbf{69.3}_{\pm0.54}66.3±0.72\mathbf{66.3}_{\pm0.72}71.3±0.65\mathbf{71.3}_{\pm0.65}50.4±0.49\mathbf{50.4}_{\pm0.49}65.1±0.67\mathbf{65.1}_{\pm0.67}42.0±0.60\mathbf{42.0}_{\pm0.60}
Reviewer_j1fsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FAME, a generalized category discovery (GCD) framework that adaptively combines frozen CLIP image features with trainable visual representations. The authors define a "cluster-level familiarity" score based on the consistency of assignments under perturbations and on prototype margins. They also define a risk score based on the disagreement between the frozen CLIP encoder and the trainable encoder and on CLIP assignment entropy. These scores route the contributions of two energy functions: one computed from the frozen CLIP features, and one from global and local representations of the trainable image encoder. Experiments cover six benchmarks under standard GCD settings. Ablations cover alternative gates, ways to stabilize the components, normalization alternatives, and other reference encoders. Appendix B gives bounds on how much the routed semantic term can shift assignment energies.

Critical strengths

S1. The motivation is well founded. Pretrained representations provide semantic knowledge, but the reliability of this evidence varies across discovered clusters, making uniform fusion potentially suboptimal. This is a clear reason to move toward a method that introduces adaptive routing between learnable visual features and guidance from a frozen pretrained model.

S2. The evaluation is broad, and the results of the main comparisons are strong. It covers alternative routing strategies, settings with known and unknown total category count, normalization variants, validation of the scores at multiple stages of training, and several reference encoders. These results are promising regarding the effectiveness of the methodological recipe the authors provide for GCD.

Critical weaknesses

W1. The role of text is unclear in multiple places of the manuscript, and the multimodal framing is ambiguous. The familiarity, risk, and energy computations from Section 4 all operate on image features and image-space prototypes and the role of the text in the manuscript is mostly unclear. For instance, Section 5.1 affirms that the text encoder encodes old-class names in textual prototypes, and Appendix A repeats that only old-class names are used with this text encoder. Neither of those explains how these features enter objectives or assignments.

Appendix C states that class names and text embeddings are not used during training, clustering, routing or model selection. Additionally, Fig. 3 adds to the confusion, showing a text encoder with the prompt "A photo of [Bird]" and shows the risk as something related to text-vision misalignment. However, Eqs. 9-10 compare two image-feature assignments. The use of DINOv2 in Tables 16 and 17 suggests that text information would not be required for the method to be used.

If the text is not used in the main experiments and only for analysis, the title and framing should refer to the reference encoder's pretraining, not suggest multimodal interaction during discovery. I would ask the authors to reconcile these descriptions and clearly state which components use text.

W2. The warm-up stage, the base method, several baselines, and the training procedure are not described in sufficient detail.

  • Section 3 states that novel prototypes are initialized "by a warm-up GCD model or clustering algorithm", Algorithm 1 (step 2) mentions a "warm-up GCD training" step, and Section 4.3 describes FAME as refining assignments "after warm-up GCD initialization", but the warm-up is never described. Is it a full GCD training (if so, which method does it build on, and for how long was it trained), or only a clustering of pretrained features?
  • Several baselines are similarly underspecified and appear inconsistent. The weight of the fixed semantic fusion baseline in Table 3 and Figs. 4–5 is not reported, nor is it discussed how it was chosen or whether it is kept constant or scheduled, as with the ramp-up coefficient of the main method.
  • The "Base GCD" in Figs. 4–5 reaches a substantially higher New accuracy on CUB, far above that of SelEx-CLIP [1] (82.682.6, 85.485.4, and 88.988.9 vs 76.576.5), which Section 5.4 refers to as the baseline. Since AL-GCD [2] is reported separately, it is unclear which method Base GCD denotes, and whether it corresponds to the warm-up method/model.
  • Additionally, the "feature-only baseline" in Table 16, described as using frozen features, has numbers identical to SelEx-CLIP in Table 1, which fine-tunes the backbone. This matters for interpreting the results, since if a full GCD model is trained first, FAME builds on its output and the comparison with prior methods should account for this.
  • It is not stated which terms of the base objective use Eq. 20, whether the routed probabilities act as predictions, self-distillation targets, or both, and whether they affect the contrastive objectives usually present in GCD methods. Since αsem\alpha_{\text{sem}} depends on a sample's current cluster, it is also unclear how test samples are routed at inference.

These details matter for interpreting the results and for reproducibility: if a full GCD model is trained first, FAME builds on its output, and the comparison with prior methods should account for this.

Overall, I find the motivation (S1) and the empirical results (S2) convincing, but they are currently outweighed by W1 and W2. The role of the text modality and the missing details about the base method, warm-up, baselines, and training procedure are central to understanding the paper's contributions and positioning. For these reasons, I currently lean towards rejection. I am open to reassessing my score depending on the authors' response.

[1] Rastegar, Sarah, et al. "SelEx: Self-Expertise in Fine-Grained Generalized Category Discovery”. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[2] Han, Jizhou, et al. "Learning Like Humans: Analogical Concept Learning for Generalized Category Discovery." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.

Questions for authors
  1. (W1) Which components, if any, use the old-class text embeddings mentioned in Section 5.1 and Appendix A, and if old-class text embeddings enter the main pipeline how do the results change when they are removed?
  2. (W2) What does the warm-up stage consist of: a full GCD training (with which method, and for how many epochs) or a clustering of pretrained features?
  3. (W2) Which methods do "Base GCD" (Figs. 4–5) and the "feature-only baseline" (Table 16) refer to? What is the "visual-only baseline" used as the reference for the gains in Tables 6, 7, 14, and 15 in the appendix? Is it the same as Base GCD?
  4. (W2) Which loss terms use the routed probabilities of Eq. 20, and how is αsem\alpha_{\text{sem}} computed for a test sample at inference?
  5. Could the authors report standard deviations over the three seeds, at least for the main results?
12 comments

Comment 1. Clarifying the Definition and Use of Text Throughout the Manuscript


We thank the reviewer for identifying this ambiguity. FAME’s default configuration is multimodal\color{brown}{\text{FAME’s default configuration is multimodal}}: it explicitly combines image features with textual anchors derived from available old-class names. After text conditioning, the downstream assignments and energy computations use a unified feature-level formulation. Our presentation emphasized these downstream operations without sufficiently explaining the upstream text-conditioning pathway. Below, we clarify the computational role of text and reconcile the descriptions throughout the manuscript.


Q1. How do old-class text embeddings enter familiarity, risk, energies, and training objectives?


① Code path and computational role.

The relevant data flow is orchestrated by run_training() in fame/engine.py:

Old-class names
  -> fame/backbone.py: FAMEModel.encode_old_names()
  -> fame/semantic.py: FAMESemanticMapper.set_basis()
  -> fame/backbone.py: FAMEModel.semantic_from_reference()
  -> fame/routing.py: reliability statistics and energy routing
  -> fame/losses.py: gcd_objective()

The frozen CLIP text encoder first encodes available old-class names into textual anchors. The mapper stores these alongside visual anchors computed from labelled old-class images. It is pretrained on labelled old examples using their textual embeddings as targets, then frozen during GCD training. For a normalized frozen reference image feature rir_i, the semantic representation is $ u_i=\operatorname{norm}\left(A_\psi(r_i;P_o^I,T_o)\right), $ where PoIP_o^I and ToT_o denote old-class visual and textual anchors, respectively. Image queries attend to these anchors to obtain text-conditioned semantic representations. These old-class anchors are distinct from the semantic cluster prototypes used for assignment, which are constructed from the resulting semantic features.

This representation contributes to three computations:

  • Familiarity and risk: FAME.accumulate_statistics() and FAME.commit_statistics() use semantic assignments and similarities to estimate stability, prototype margins, assignment disagreement, and entropy.
  • Energy and assignment: FAME.energies() computes the semantic energy from these features and their cluster prototypes; FAME.forward() combines semantic and visual energies to produce routed predictions. Text therefore affects Eq. 14\color{blue}{\text{Eq. 14}} and Eqs. 17–20\color{blue}{\text{Eqs. 17–20}}.
  • Training objectives: gcd_objective() uses routed logits for labelled old-class classification, both the teacher and student sides of cross-view self-distillation, and prediction balancing. The contrastive terms operate on projected visual features without an explicit text-similarity term.

We will replace the raw-image-only reference-feature definition in Section 4\color{blue}{\text{Section 4}} with the text-conditioned definition above, clarify its use in Section 5.1\color{blue}{\text{Section 5.1}}, and document mapper construction and pretraining in Appendix A\color{blue}{\text{Appendix A}}.


② Controlled text-removal ablation.

To isolate explicit text conditioning from CLIP pretraining and mapper capacity, the relevant controls are:

  • A: Visual-only base: zero semantic contribution to the routed predictions.
  • B: CLIP image reference: retain routing but use the raw frozen CLIP image feature, ui=riu_i=r_i, bypassing the text-conditioning pathway.
  • C: Visual-anchor mapper: retain the mapper architecture while replacing textual anchors and mapper-supervision targets with old-class visual anchors.
  • D: Text-conditioned FAME: use the old-class textual anchors and text-conditioned representation described above.

\color{blue}{\text{Table R1. Text-conditioning ablation. Accuracy (All / Old / New, \\%).}}

ConfigurationCUBCarsAircraftHerbarium19
A: Visual-only base82.4 / 79.3 / 84.477.3 / 91.1 / 70.464.6 / 64.0 / 64.946.4 / 60.0 / 39.6
B: CLIP image reference84.6 / 79.6 / 87.279.0 / 91.4 / 72.866.8 / 65.0 / 67.748.0 / 61.0 / 41.5
C: Visual-anchor mapper84.6 / 79.3 / 87.879.4 / 91.7 / 73.367.1 / 65.3 / 68.048.4 / 61.3 / 42.0
D: Text-conditioned FAME86.3 / 79.8 / 88.980.9 / 92.0 / 75.468.7 / 66.1 / 70.049.8 / 62.0 / 43.7

The pattern in Table R1\color{blue}{\text{Table R1}} favors explicit old-class text conditioning, with controls for CLIP pretraining and mapper capacity.

Q2. How do we reconcile the manuscript descriptions and the image-only extensions?


① Scope of the post-hoc restriction.

The statement in Appendix C\color{blue}{\text{Appendix C}} is overly broad. It does not distinguish the old-class text used in the pathway described in Q1 from additional post-hoc queries involving novel-class names and hidden labels of unlabelled examples.

For example, an available old bird-species name supplies a semantic anchor during discovery, whereas a withheld novel bird-species name is queried only afterward to evaluate cluster recognizability. Novel-class names and hidden labels do not enter training, clustering, routing, or model selection.

We will correct Appendix C\color{blue}{\text{Appendix C}} and the caption of Table 4\color{blue}{\text{Table 4}} to state this distinction explicitly. We will also distinguish the raw CLIP image prototypes used for post-hoc evaluation from the text-conditioned prototypes used for discovery.


② Correspondence between the figure and assignment disagreement.

With the reference representation defined in Q1, Eqs. 9–10\color{blue}{\text{Eqs. 9–10}} compare text-conditioned semantic and trainable visual assignments. Both branches process image inputs, but the semantic reference additionally incorporates old-class textual information.

For example, assigning the same image to cluster 2 in the semantic branch and cluster 5 in the visual branch gives a disagreement indicator of 1; matching assignments give 0. These indicators are aggregated at the cluster level and combined with semantic assignment entropy to estimate risk.

Accordingly, “Text & Vision disaligned” in Fig. 3\color{blue}{\text{Fig. 3}} describes inconsistency between text-conditioned semantic guidance and learned visual evidence. It does not denote a direct text–image embedding comparison. We will clarify this meaning and the restriction of prompts to available old-class names in the figure caption. With the reference-feature definition corrected as described in Q1, the algebraic forms of Eqs. 9–10\color{blue}{\text{Eqs. 9–10}} can remain unchanged.


③ Scope of the image-only extension.

The routing mechanism can also operate with an image-only reference. For example, the DINOv2 setting described in Tables 16 and 17\color{blue}{\text{Tables 16 and 17}} supplies frozen visual reference features without text conditioning. This examines the generality of reliability-aware routing, whereas the text-conditioned configuration described in Q1 explicitly combines textual and visual information.

We will distinguish the multimodal configuration from this image-only extension and specify whether text conditioning is active for each reference-encoder setting.

Comment 2. Clarifying the Training Pipeline, Baseline Definitions, and Inference Procedure


We thank the reviewer for identifying these missing details and inconsistencies.

Q1. What exactly does the warm-up stage consist of?

FAME starts directly from pretrained CLIP. Its default training procedure follows the workflow below. Firstly, Preparation: CLIP initialization → old-class semantic mapper pretraining (2,000 steps) → freeze the reference encoders and mapper. Next, Main procedure: ① Visual representation warm-up (20 epochs) → ② Assignment and prototype initialization → ③ FAME refinement (200 epochs). The complete procedure is orchestrated by run_training() in fame/engine.py, with default settings specified in train.py.


① Visual representation warm-up: 20 epochs.

Labelled and unlabelled images → two augmented views → contrastive representation learning → adapted visual features.

We optimize the visual branch using

Lwarm=0.65LInfoNCE+0.35LSupConL_{warm}=0.65L_{InfoNCE}+0.35L_{SupCon}

The instance-contrastive term uses two views of each image, while the supervised contrastive term uses only available old-class labels. We update the last visual transformer block, residual adapter, and contrastive projector. The reference encoders and pretrained semantic mapper remain frozen. Prototype-based classification, self-distillation, and semantic routing are inactive during this stage. This is an internal representation-learning warm-up from CLIP initialization; it does not require a separately trained SelEx-CLIP or AL-GCD checkpoint.

Code locations: fame/engine.py (train_epoch), fame/losses.py (contrastive_losses and gcd_objective), and fame/backbone.py (FAMEModel).


② Assignment and prototype initialization.

Warm-up visual features → initial clustering → old-class slot alignment → constrained clustering updates → initial memberships and prototypes.

We extract structural visual features from the training set and initialize category slots using MiniBatchKMeans. Old-class slots are aligned with the means of labelled old-class features. We then perform 20 cosine-clustering updates, keeping labelled old samples assigned to their known slots. The resulting memberships initialize the semantic, local, and structural prototypes. Thus, this stage clusters features adapted during warm-up, rather than only the original frozen CLIP features. Ground-truth labels of unlabelled examples are not used.

Code locations: fame/engine.py (collect_features), fame/initialization.py (initialize_assignments), and fame/routing.py (FAME.initialize).


③ FAME refinement: 200 epochs.

Previous assignments and reliability statistics → routed energy computation → visual-branch optimization → assignment refresh → EMA updates of scores and prototypes.

Starting from the initialized memberships and prototypes, we enable FAME routing and continue training the visual branch for 200 epochs. Each epoch uses the preceding assignments and reliability statistics for routing. After the gradient-based updates, we refresh assignments and update the cluster-level scores and prototypes. Warm-up and refinement share one continuous cosine learning-rate schedule, giving a default visual training budget of 220 epochs. The separate 2,000-step mapper pretraining precedes these stages and is documented in fame/semantic.py (pretrain_mapper). We will report the complete computational cost, including mapper pretraining, visual warm-up, clustering initialization, and refinement.

Code locations: fame/engine.py (run_training, train_epoch, and refresh_clusters), fame/routing.py, and fame/losses.py.


We will clarify these specific steps and their execution order in Appendix A\color{blue}{\text{Appendix A}}.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.