acceptodds
Under review as a conference paper at ICLR 2027

Learning New Classes Federatively without a Shared Label Space

Abstract

Federated few-shot class-incremental learning (FFSCIL) assumes that new classes arrive in one shared global label space. Once clients label the world with private, partially overlapping taxonomies, the server must first decide which client classes denote the same concept, from one to five samples and with no correspondence oracle. We study this concept-identity problem as Taxonomy-Free FFSCIL and build its benchmark from real taxonomies: synonyms, real homonyms, and clients that label at the superclass level. We propose FedTaxo, which encodes each local class as a feature-sharing concept capsule and grows a latent concept graph by re-clustering every novel capsule each session under the constraint that two identifiers of one client are never one concept. For a Gaussian surrogate, a centered threshold identifies concepts at the minimax rate, and constrained re-clustering recovers a session's exact partition under a separating score. Controlled experiments on four datasets show that the assignment procedure matters more than the matching score: re-clustering recovers 85% of an oracle's novel-class gain, against 69% when the same score is assigned greedily, while a learned matcher adds little accuracy but recovers far more of the true sub/super edges. When client taxonomies genuinely cross-cut, even merging by true set overlap falls below not federating. Code, protocol and LLM-written client vocabularies are available at https://anonymous.4open.science/r/fedtaxo-iclr27-FF2A/.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
federated few-shot class-incremental learning, private label spaces, cross-client concept alignment, constrained revisable clustering, label synonymy and homonymy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DE5iOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Taxonomy-Free Federated Few-Shot Class-Incremental Learning (FFSCIL), addressing the challenge of aligning private and partially overlapping class taxonomies across clients without a shared label space. It proposes FedTaxo, which represents local classes as concept capsules and uses constrained, revisable clustering to construct a shared concept graph while preserving client-specific labels. The paper also provides theoretical analysis of concept matching and assignment recovery. Experiments on four benchmarks demonstrate that constrained re-clustering recovers approximately 85% of the oracle's novel-class improvement, outperforming greedy assignment and text-based alignment methods.

Critical strengths
  1. The paper introduces Taxonomy-Free FFSCIL, addressing the realistic challenge of inconsistent and partially overlapping class taxonomies across federated clients. This extends conventional FFSCIL beyond the shared-label-space assumption.
  2. FedTaxo employs constrained, revisable clustering to align local concepts across clients. The theoretical analysis provides additional justification for the proposed assignment strategy.
  3. Experiments on four benchmarks demonstrate substantial improvements over existing alignment strategies. The controlled comparisons and ablation studies further highlight the importance of assignment procedures over matching scores.
Critical weaknesses
  1. The experiments rely primarily on simulated client taxonomies rather than independently constructed real-world label systems. This limits the evidence for practical applicability in realistic federated environments.
  2. FedTaxo relies on within-client cannot-link constraints and compatible concept hierarchies. Its performance degrades when these assumptions are violated, particularly under duplicate labels and cross-cutting taxonomies, raising concerns about generalizability.
Questions for authors
  1. Clarify how well the simulated taxonomy settings reflect real-world federated environments and discuss the expected performance when clients use independently developed label systems.
  2. Explain the sensitivity of FedTaxo to violations of within-client cannot-link constraints and cross-cutting taxonomies, and clarify whether the proposed soft-constraint strategy can effectively mitigate these limitations.
1 comment
AuthorsOpenReview ↗

Response to Reviewer DE5i

Response to Reviewer DE5i:

Thank you for the review. The revised PDF marks all changes in blue. The main addition is a federation of real, separately built label systems (new Sec. 4.5, Table 4); Appendix H holds every new experiment. An evidence map after the appendix contents lists the tables behind each research question.

W1 / Q1. Real, independently developed label systems. Each client in the new federation is a real remote-sensing scene dataset with its own class list. The seven datasets are UC Merced, NWPU-RESISC45, AID, PatternNet, RSI-CB256, WHU-RS19 and RSSCN7 (7 to 45 classes), built by teams at four universities. Clients keep their original class names. Nothing is renamed or corrupted, and there is no shared base session and no labeled reference. Every client is scored on its own test images in its own label space.

Avg minus not federating (paired pointwise 95% CI, Table 41)5-shot1-shotpair F1 (5 / 1)
exact name match+0.03 ± 0.23+2.87 ± 0.3974 / 74
text-NN-13.70 ± 2.97-8.13 ± 2.2339 / 42
FedTaxo-1.59 ± 0.13+2.08 ± 0.4687 / 80
oracle-0.48 ± 0.21+3.99 ± 0.85100

How well does the simulator reflect this? Three differences matter.

  1. Exact names are reliable across these datasets: an identical name always meant the same concept. Our simulated homonyms are harsher, so the simulator understates exact name matching.
  2. Real synonyms and near-collisions (parking lot/parking space, storage tank/storage room) are harder for text alignment than our simulated ones. Text-NN loses 8 to 14 Avg here.
  3. The simulator has no shift between clients' images, and real organizations do. At five shots this outweighs the benefit of extra shots, even for the true correspondence.

Expected performance. FedTaxo recovers the correspondence best. It gains 2.1 Avg at one shot (half of the oracle's 4.0), while every other similarity-based rule loses 8 to 26; exact name matching gains slightly more (+0.8 ± 0.4 over FedTaxo). At five shots neither the similarity-based rules nor the oracle improve on not federating: FedTaxo loses 1.6, the oracle 0.5, other similarity rules 5 to 14, and name matching ties. A new client-side check (below) brings every rule but discovery to within 0.11 of not federating there; it reduces negative transfer but does not guarantee none. We revised the abstract, contributions and conclusion to state these mixed results plainly.

W2 / Q2. Violated assumptions. Your comment covers two assumptions, so we answer them in turn.

(a) Duplicate labels break the within-client cannot-link. When a client holds a duplicate label for a concept with probability p = 0.2 or 0.4, FedTaxo loses 2.1 or 3.3 Avg while the oracle gains (Table 29). The soft rule lets a client's two labels merge above a raised threshold. It is the best of hard, soft and no constraint in five of six cells, and always beats dropping the constraint. It recovers only part of the loss, though: 36 and 38% on CIFAR-100 and 59 and 53% on CUB-200 (Table 31; these results were already in the submission). So it mitigates the problem but does not solve it. We also tested the new client check here (Appendix H.6, Table 48). It does not help, because the harm is under-merging, not a wrong merge. The paper names duplicate labels as the method's main exposure.

(b) Cross-cutting taxonomies break hierarchy compatibility. When clients partition the same classes in incompatible ways, no method gains consistently (Table 12; Appendix D.2). FedTaxo stays close to not federating on all four datasets (79.25 vs. 79.18, 92.71 vs. 92.76, 83.65 vs. 83.44, 57.98 vs. 57.75). Text alignment loses up to 3.0, and merging by true class overlap up to 1.8. The new client check holds out each of a client's own shots and keeps the merged prototypes only if they do not lower its classification margins. With it, FedTaxo matches or beats not federating on all four datasets, and text-NN rises to within 0.02 of not federating. In the default and coarse-label regimes the check keeps at least 99% of FedTaxo's merges, so it costs almost nothing where alignment helps. We state cross-cutting taxonomies as a boundary of the formulation: a concept graph whose nodes are concepts cannot represent overlapping, non-nested partitions.

Other additions, in brief.

  • FedTaxo's threshold can be calibrated without a shared labeled reference or correspondence annotations, from scores of a client's own label pairs. It stays within 0.2 Avg of the labeled dry run in seven of eight settings (Table 44).
  • In our simulator, the re-clustering gain comes from assigning each session's capsules jointly. Revising earlier sessions matters on the real federation (+2.2 ± 0.9 Avg; Tables 41 to 43).
  • We report membership inference against capsules (Table 47), the cost growth with sessions (Table 49), and a learner that keeps updating its prompt (Table 50).
Reviewer_xANUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates federated few-shot class-incremental learning without assuming a shared label space across clients. Different clients may use different names for the same concept or the same name for different concepts. To address this issue, the authors propose a taxonomy-free framework that identifies and aligns concepts using uploaded class representations and a constrained clustering procedure. The paper also provides theoretical analysis and constructs benchmarks based on WordNet taxonomies and simulated client-specific vocabularies.

Critical strengths

Existing federated class-incremental learning methods generally assume that class identities are consistent across clients. This assumption may not hold in practical applications, especially when different organizations maintain their own labeling systems. The authors identify this issue and provide a systematic formulation. I consider this a meaningful extension of the existing federated learning setting. The proposed concept alignment framework is reasonably designed. In particular, the constrained clustering procedure allows previously established concept assignments to be revised as new information becomes available. This is useful for incremental learning, where early matching errors may otherwise accumulate over time. The theoretical analysis provides further motivation for the problem. The controlled experiments separating matching scores from assignment procedures are also helpful in understanding where the improvement comes from. the paper addresses an important problem that has received relatively little attention.

Critical weaknesses

The main limitation is that the evaluation relies on simulated client taxonomies. Although the benchmark construction is more realistic than simply perturbing class names, it remains unclear how well the proposed method would work with independently developed labeling systems from real organizations. The performance improvements are also modest in some settings (typically 0.7–1.2 points on the main metric). The margins are thin. More discussion is needed on when cross-client concept alignment is beneficial and when it may introduce additional errors. Another concern is privacy. The uploaded concept capsules contain sample-level feature embeddings, which may expose information about local data. The current experiments do not provide sufficient evidence regarding potential information leakage. Because privacy is the fundation of FL. In addition, the cannot-link constraint assumes that different local labels correspond to different concepts. This assumption may be violated when a client uses multiple labels for the same underlying concept. The paper discusses this case, but the proposed relaxation does not fully resolve the problem (it recovers only 38–53% of the resulting loss).

Questions for authors

Have the authors considered evaluating the method on real datasets collected from different organizations with independently defined label systems? Could the authors clarify the privacy threat model? In particular, can the uploaded sample embeddings be used for feature inversion or membership inference? How does the computational cost of concept re-clustering increase with the number of incremental sessions and accumulated concepts? Is it possible to identify, before federation, whether a particular client taxonomy configuration is likely to benefit from concept alignment?

1 comment
AuthorsOpenReview ↗

Response to Reviewer xANU

Response to Reviewer xANU:

Thank you for the careful review. The revised PDF marks all changes in blue. The main additions are a federation of real, separately built label systems (new Sec. 4.5, Table 4) and Appendix H, which holds every new experiment. An evidence map after the appendix contents lists the tables behind each research question. Intervals are paired, pointwise 95% CIs over five seeds (Table 41); every effect in the table below except name matching at five shots survives Holm correction.

W1 / Q1. Real label systems from different organizations. Each client is a real remote-sensing scene dataset with its own class list: UC Merced, NWPU-RESISC45, AID, PatternNet, RSI-CB256, WHU-RS19 and RSSCN7 (7 to 45 classes, 195 in all), built by teams at four universities. Clients keep their original names, so the synonyms (highway/freeway, port/harbor), near-collisions (parking lot/parking space) and coarse labels are real. There is no shared base session or labeled reference, and each client is scored on its own test images.

Avg minus not federating5-shot1-shotpair F1 (5 / 1)
exact name match+0.03 ± 0.23+2.87 ± 0.3974 / 74
text-NN-13.70 ± 2.97-8.13 ± 2.2339 / 42
FedTaxo-1.59 ± 0.13+2.08 ± 0.4687 / 80
oracle-0.48 ± 0.21+3.99 ± 0.85100

(Not federating: 92.52 and 78.91 Avg.) The results are mixed, and we report them as such. At one shot FedTaxo gains 2.1 Avg and recovers the correspondence best, while every other similarity-based rule loses 8 to 26 points. Exact name matching gains more (+0.8 ± 0.4 over FedTaxo): here an identical name always meant the same concept, unlike our simulated homonyms. At five shots neither the similarity-based rules nor the oracle improve on not federating, and name matching ties.

W2. Thin margins, and when alignment helps. 60 of 100 classes (100 of 200 on CUB) are base classes that every front-end treats alike, which compresses Avg. On novel classes FedTaxo gains 5 to 32 points over not federating, and 2.4 to 11.5 over text-NN in seven of eight settings (Table 2; five-shot CUB is -1.0). Alignment helps when a concept reaches several clients and the within-client constraint holds. It hurts under cross-cutting taxonomies, duplicate labels, and, on real data at five shots, when organizations' images differ enough that even true merges hurt.

Q4. Knowing in advance whether alignment will help. We do not establish a reliable predictor before federation. What we added is a check a client runs after the server proposes merges and before it adopts them (Appendix H.6, Table 48). It holds out each of its own shots and keeps the merged prototypes only if they do not lower its classification margins. It keeps at least 99% of FedTaxo's merges where they help. Under cross-cutting taxonomies it lifts text-NN to within 0.02 of not federating; on the real federation at five shots it brings every rule but discovery to within 0.11. It reduces negative transfer but does not guarantee none (FedTaxo: 92.43 vs. 92.52). It needs two shots per label, so at one shot it accepts everything.

Q2. Privacy. Threat model: an honest-but-curious server sees every upload. FedTaxo needs only the prototype, shot count, label string and a scalar spread; withholding the shot embeddings leaves every result identical (only FedTaxo-L needs shots). Membership inference against same-class non-members (Appendix H.5, Table 47):

  • Uploaded shots are identified exactly (AUC 1.00).
  • A five-shot prototype still leaks (AUC 0.76 to 0.78); a one-shot prototype is the shot itself.
  • Noise of sigma = 0.2 brings five-shot AUC to 0.54 (epsilon about 10) but removes FedTaxo's gain. At one shot the same noise leaves AUC at 0.71 to 0.73.

We found no setting that both protects the shots and keeps the gain, and we did not evaluate feature inversion. The paper states that capsules are feature sharing without a formal guarantee.

Q3. Cost as sessions accumulate. Re-clustering scores every pair of units, so cost grows about quadratically: on CUB features with 10 clients, 0.55 s per session at 300 accumulated capsules and 11.4 s at 1,200 (Table 49). A variant that never revises earlier sessions scores only pairs with a new capsule (1.8 s at 1,200). It matches full re-clustering in our simulator but loses 2.2 Avg on the real federation, where revision matters.

W4. Duplicate labels. We agree the relaxation does not resolve this; that result was already in the submission. Duplicates cost FedTaxo 2.1 and 3.3 Avg at p = 0.2 and 0.4 while the oracle gains. The soft rule recovers 36 to 59% of the loss (Table 31). The new client check does not help, because the harm is under-merging. The paper names this as the method's main exposure.

Reviewer_rWzoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies federated few-shot class-incremental learning with unaligned client labels. FedTaxo combines visual and textual concept capsules with constrained re-clustering and a graph readout in each client's label space. The theoretical analysis gives identifiability and recovery conditions under explicit surrogate assumptions. Experiments on simulated client taxonomies compare matching scores and assignment procedures, assess hierarchy recovery, and test departures from the default setting.

Critical strengths
  1. The paper clearly distinguishes local identifiers from label descriptions. Evaluating predictions in each client's own label space makes the alignment problem concrete and avoids assuming away the mismatch being studied.

  2. The method is simple and well specified. Retaining capsules allows assignments to be reconsidered, while the graph readout supports coarse and fine labels without imposing a shared output vocabulary.

  3. The score-by-assignment comparison directly tests the main empirical claim. The broader evaluation considers novel-class prediction and recovered relations separately, and the ablations investigate client constraints and matcher features.

  4. The paper reports adverse results and explains their implications. The boundary tests and the discussion of classifier transfer make the empirical scope easier to assess.

Critical weaknesses
  1. The setting still requires aligned reference supervision for calibration. As Sec. 4.4 explicitly notes, removing the canonical base graph does not remove the canonically labeled base data used to select the threshold. The demonstrated setting therefore requires an aligned labeled reference, even when the deployed base graph is unaligned.

  2. The trained-classifier comparison in Sec. 4.2 uses a base-trained prompt model with fixed representations during novel sessions, but the Fed-L2P experiment does not establish useful alignment gains, even with oracle correspondence. Transfer to incremental learners with continuing parameter updates therefore remains unverified.

Questions for authors
  1. For Table 1, do the existing runs distinguish corrections to earlier assignments from improvements in joint assignment within the current session?

  2. For Table 3, is a comparison available that holds the learned partition fixed and disables only descendant-label inheritance?

2 comments
AuthorsOpenReview ↗

Response to Reviewer rWzo

Response to Reviewer rWzo:

Thank you for the precise questions. They led to two corrections of our own claims and one clarification. The revised PDF marks all changes in blue; Appendix H holds every new experiment. An evidence map after the appendix contents lists the tables behind each research question. Intervals below are paired, pointwise 95% CIs over five seeds; significance claims use one-sided paired t-tests with Holm correction.

W1. Calibration needed an aligned labeled reference. We removed that need for FedTaxo. Two identifiers of one client are known to be different concepts, so their scores are negatives that need no cross-client annotation. The new rule sets tau, each session, to the 0.95 quantile of these scores (q fixed before running). It needs no dry run, no shared labeled reference and no correspondence annotations. It still uses each client's own labels and the cannot-link, and FedTaxo-L's matcher still trains on the labeled base session.

  • Against the labeled dry run it stays within 0.2 Avg in seven of eight settings (Table 44).
  • With the private base session, no canonical or cross-client label is used; FedTaxo stays within 0.32 Avg of the aligned protocol (Table 45).
  • The real federation (Sec. 4.5) has no base session and no shared reference.

The threshold matters little in our simulator because each concept reaches three of five clients: two complete clusters always share a client, so the cannot-link forbids their merge. Correction: the submission said the dry run picked tau = 0.3 everywhere; for the cosine it picked 0.7 in six of eight settings (Sec. 3.2).

W2. A learner whose parameters keep changing. In every novel session each client fine-tunes the shared prompt on its new shots. The loss is cross-entropy over a cosine head with one row per concept node (the global label space), plus distillation to the previous prompt; the server applies FedAvg. Every capsule is then re-encoded. An alignment error thus becomes a training label (Appendix H.8, Table 50).

Avg (%), CIFAR-100K=5: not fed. / FedTaxo / oracleK=1: not fed. / FedTaxo / oracle
frozen prompt86.03 / 86.82 / 86.8381.23 / 84.54 / 85.07
mild updates85.99 / 86.70 / 86.7081.32 / 84.55 / 85.05
strong updates85.88 / 86.42 / 86.4281.34 / 84.49 / 84.95

Features move little (1 - cos at most 0.04) and no front-end gains accuracy, so this tests robustness to a moving representation. The ordering holds: FedTaxo matches the oracle at five shots and stays 0.5 to 3.3 above not federating; text-NN stays below it at five shots. The scope is one dataset and one architecture; we make no claim for learners that retrain the backbone.

Q1. Corrections of earlier assignments vs. joint assignment. The submitted runs did not separate them. The new past frozen variant starts each session from the previous partition and never splits or joins earlier nodes; we also count earlier pairs each session fixes or breaks (Tables 42 and 43).

  • Cosine score: freezing matches or beats full re-clustering in all eight settings. It is significantly better only at one shot on CIFAR-100 and CUB-200 (Holm over eight tests, adjusted p = 0.004 and 0.010; Aircraft's interval excludes zero but p = 0.076). It beats greedy assignment by 0.81 Avg on average.
  • Learned score: revision helps on five-shot Aircraft (+0.10 ± 0.06, adjusted p = 0.039).
  • Real federation, where concepts reach clients at different times: revision is worth 2.2 ± 0.9 Avg at five shots and 2.2 ± 1.1 at one.

Relation to Table 1. The cosine re-clustering column equals Table 1 exactly. Table 42's greedy cosine runs the re-clustering code greedily with the same threshold grid, so only the assignment differs; Table 1's FedTaxo-C selects its threshold among F-beta candidates only. They differ only on one-shot miniImageNet (89.75 vs. 89.43). The learned-score runs use a fixed hash seed, on which FedTaxo-L's simulator depends, so they differ from Table 1 by up to 0.46. Appendix H.2 states this. Clarification: Table 1 therefore compares methods, not assignment alone. With matched grids (Table 42) re-clustering is worth +0.66 Avg (not +0.70), and the learned matcher +0.36 under greedy assignment (not +0.47); Figure 1, Sec. 4.1, 4.2 and the contributions now say so.

Correction: the submission credited the gain to revisability. The paper now credits joint assignment in our simulator, and revision where arrivals are spread over time.

Q2. Fixed partition, inheritance off. Table 46 scores the same graphs with and without inheritance (fixed hash seed; up to 0.7 from Table 3). FedTaxo-L's coarse accuracy falls from 67.5 to 64.1 on CIFAR-100 and from 80.5 to 75.3 on CUB-200, against 58.0 and 69.5 for not federating. Inheritance is worth 3.4 and 5.2 points; the partition alone gives +6.1 and +5.8. On miniImageNet and Aircraft it changes at most 0.4. For the oracle it is worth 6.5 to 12.3.

Thank you for the careful response and the additional controls. The revised manuscript addresses my questions about reference supervision, continued prompt updates, retrospective revision and descendant inheritance. I also appreciate the correction separating joint assignment from revision across sessions.

While the real federation provides valuable evidence, it also shows that better concept alignment does not consistently improve prediction under client distribution shift. The continued learning result supports robustness to modest prompt drift within the tested setting. These limits constrain the reach of the contribution, although I regard the problem formulation and the more precise experimental diagnosis as worthwhile.

Overall, I have no further questions on the issues raised in my original review, and remain at weak acceptance.

Comments on the paper

1 comment
Area_Chair_3KnDOpenReview ↗

Urgent Reminder: Reviewer Discussion Deadline

Dear Reviewers,

This is a friendly but urgent reminder that the discussion period will end on November 18, 2026.

Please carefully review the authors' rebuttals and actively participate in the ongoing discussions. If you have not yet responded to the authors or initiated a discussion, please do so as soon as possible.

In particular, reviewers are expected to address the authors' clarifications and responses, provide constructive follow-up feedback, and update their assessments and scores when appropriate.

Please ensure that your final evaluations reflect your independent judgment and take into account the information provided during the rebuttal and discussion process.

Active and timely participation is essential to ensuring a fair, thorough, and constructive review process.

Thank you for your time, effort, and cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.