- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 3
Read review
Paper summary
The authors address a practical issue in federated learning known as "fragmentary recurring concept drift": the complete label mapping currently held by a client may be novel, yet the constituent local mapping rules have already been learned by other clients or in previous rounds. To tackle this, the authors propose FragMem, a method where the server maintains a persistent "label-state" memory bank. Clients use feature statistics from their current labeled batches to match historical states, retrieving classifier rows for specific labels to assemble a classification head; following local training, the server aggregates updates based on label and state slots, enabling the reuse of local knowledge across clients and time. Evaluations on three constructed drift benchmarks show that FragMem performs close to an oracle that knows the true label states and significantly outperforms a baseline using whole-model memory. Extended experiments further analyze the benefits and failure boundaries under conditions such as many-to-one mapping, label sparsity, class imbalance, real-world sentiment grouping, and strong feature representations.
Critical strengths
-
The authors propose two key concepts in federated learning—"reproduction of complete concepts" and "formation of new combinations from known fragments"—and point out that the storage granularity of historical knowledge limits its reusability. This offers a valuable research perspective on federated concept drift.
-
The experimental analysis is comprehensive, covering aspects such as H1 negative controls, changes in recombination requirements, and recovery curves following the switch. Additionally, the study reports experiments addressing label sparsity and imbalance and clearly delineates the boundaries between theoretical assumptions and statistical analysis, validating the method's robustness.
Critical weaknesses
- The concept proposed by the authors appears quite similar to TTA for FedDG and FedProto; I believe the authors should compare their approach with these methods in the experiments and clarify the boundaries and distinctions in the "Related Work" section.
2.The authors did not conduct tests on large-scale ViT models; given that current models are predominantly Transformer-based, thorough validation on this architecture is required.
Questions for authors
- I believe the framework diagram in the paper needs to be redrawn; although I have provisionally given a positive rating, I hope the authors will treat the submission seriously. I will lower the score if the author does not address my questions.
- Refer to the issues mentioned in the weakness.
6 comments
Author Response (2/3)
Validation on large-scale Vision Transformers (DINOv2 ViT-S/14, ViT-B/14, and ViT-L/14) and full 50k/10k CIFAR-100
W2: The authors did not conduct tests on large-scale ViT models; given that current models are predominantly Transformer-based, thorough validation on this architecture is required.
Thank you for pointing out the need for prominent Vision Transformer validation—our initial submission placed DINOv2 in an appendix table and omitted larger ViT backbones. First, our submitted Table 4b already evaluated frozen DINOv2-small (ViT-S/14) on CIFAR-100 H2: with an MLP head, FragMem reaches 84.2% (matching Class-identity oracle (*)), leading fine-tuned FedAvg+FT (80.9%) by +3.23 pp, FedBABU (76.4%) by +7.7 pp, FEROMA-τ (65.6%), and Concept-unit control (62.7%), and reaches 83.6% with a linear head (95% CIs in main-text Table 3b, Section 5.2, p. 8).
Second, to validate larger Transformer encoders at full dataset scale with strictly zero test leakage, we completed new experiments on full 50,000/10,000 CIFAR-100 using frozen DINOv2 ViT-B/14 (both superclasses and fine classes with blocks and batch size 500) and DINOv2 ViT-L/14 (), projected with strictly inductive train-only PCA-128 fit on outer training splits alone (hyperparameters locked on 80/20 training-split holdouts across seeds 42–43, confirmed on seeds 44–48):
Full 50k/10k Pool (H2, T0/T*) | FragMem | BestWhole (oracle wr.) | Concept-unit ctrl (best-loss) | FedAvg | FedProto-LocalHead | Class-identity oracle (*) |
|---|---|---|---|---|---|---|
DINOv2 ViT-B/14 () | 91.91/91.96 | 71.39/89.09 | 71.35/89.05 | 35.51/88.64 | 69.86/88.44 | 91.93/91.98 |
DINOv2 ViT-L/14 () | 94.53/94.57 | 72.65/92.09 | 72.62/92.08 | 36.63/91.87 | 71.21/91.69 | 94.53/94.57 |
DINOv2 ViT-B/14 (, ) | 85.75/86.33 | 77.79/84.02 | 77.06/83.39 | 31.08/82.29 | 75.15/81.89 | 86.02/86.57 |
Across all three large-scale ViT benchmarks, FragMem delivers consistent gains across the 5 confirmation seeds even when stronger Transformer features raise overall accuracy ( in Table 12, p. 29):
- Consistent margins over whole-head and fine-tuned baselines. At development-selected ,
FragMemleadsBestWhole (oracle write)by+2.87 / +2.48 / +2.31pp andFedAvgby+3.32 / +2.70 / +4.04pp across the three pools. Even unadaptedFragMem(T0)leads fine-tunedBestWhole(T*)by+2.82 / +2.44 / +1.73pp (fullNoMem-RidgeCVandLossMatchscores appear in Table 12). - No negative transfer on
H1and89.96xbandwidth savings at . On theViT-B/14H1seen-concept control, the clean comparison at unadaptedT0is+0.39pp (dev-selectedT*slightly lowersConcept-unit controlfrom91.58%to91.06%, wideningT*to+0.92pp). At fine classes, transmitting whole-head candidate pools consumes37,277.65MB compared to414.40MB forFragMem(89.96xsavings).
We placed Table 3b in Section 5.2 (p. 8) and added these full-data ViT-B/14 and ViT-L/14 results to Section 5.7 (p. 10) and Appendix I (Table 12, p. 29) of the revised manuscript.
Author Response (3/3)
Redrawing Figure 2 into a three-lane, five-step protocol diagram
Q1: I believe the framework diagram in the paper needs to be redrawn; although I have provisionally given a positive rating, I hope the authors will treat the submission seriously. I will lower the score if the author does not address my questions.
Thank you for raising this directly—we take your feedback very seriously. Looking back at the submitted Figure 2, we agree that packing client updates, server lookups, and round timing into one dense diagram obscured both data isolation and execution order.
We have therefore completely redrawn Figure 2 in the revised manuscript (Section 4, p. 5, with caption revisions highlighted in red) as a three-lane vector diagram (Client k, Server, and Persistent Slot Bank) that separates each round into five explicit steps synchronized with Section 4 notation:
- Step 1 — Batch Isolation (
obs. c vs. state s_c). TheClient klane explicitly separates frozen representation over the disjoint round- training batch (with observed labelsobs. cmapped from latent statestate s_c) and held-out evaluation batch . - Step 2 — Moment Upload. From alone, the client computes and uploads per-label counts , means , within-label second moments , and split-half statistic for calibrating novelty threshold .
- Step 3 — Slot Match / Spawn & Assembly. Without any oracle state IDs, Eq. (1) matches each observed label against stored descriptors and rows in the persistent
(label, slot)bank, selects or spawns active slot index , and splices the assembled head . - Step 4a — Pre-Train Serving. The assembled model (strictly from historical writes ) predicts first, visually separating evaluation from local training; the optional client-side
+FTcontrol on is drawn on a dotted auxiliary clone branch inside theClient klane. - Steps 4b–5 — Persistent Writeback. Only after evaluation do clients run persistent 10-epoch SGD on and upload updated classifier rows and feature moments, which the server aggregates strictly by
(c, j)slot key back into the persistent bank for round .
The diagram in Section 4 (p. 5) and its walkthrough in Appendix C (p. 17, which groups Steps 1–2 under its first bullet) now map step-by-step in our revised PDF.
Author Response (1/3)
We thank the reviewer for the positive rating, for recognizing our fragment-versus-whole-concept storage distinction, and for clear guidance on scope and presentation. Across three comments—integrated into our companion revised manuscript with new and substantively revised passages, figures, and tables highlighted in red (plus a full latexdiff PDF and a completely redrawn Figure 2 on page 5)—we address (1)
FedProtoand federated domain generalization (FedDG) / test-time adaptation (TTA) (W1,Q2), (2)DINOv2ViT-S/B/L validation (W2), and (3) the redrawn Figure 2 (Q1).Experimental comparisons and conceptual boundaries relative to
FedProtoandTTA/FedDGThank you for highlighting these comparisons.
Table 6in the initial submission (Table 7, Appendix H, p. 23of the revised manuscript) already reports unadaptedFedProtoonH2(38.4 / 53.8 / 36.4%local model and20.3 / 23.2 / 25.1%global prototype inference vs.63.1 / 86.6 / 64.4%forFragMem). To compare fine-tunedFedProtovariants and a Tent-style linear-head entropy minimization probe (FedAvg-LinEntropy) under matched round-t access (20 clients, T=100 rounds including a 20-round burn-in on fixed 1,920/480 splits;H2: unseen recombinations;T0: 0 steps; best tierT*locked on seeds 42–43 and confirmed on seeds 44–48), each method tunes a disposable head copy on Dtraink,t (0, 5/20 full-batch steps, or 1/2/5 50-sample mini-batch epochs) before predicting Devalk,t (FedAvg-LinEntropyinstead minimizes (W,b) prediction entropy directly on unlabelled Devalk,t, with zero persistent writeback).H2switch %)T0/T*)T0/T*)T0/T*)FragMem - Method @ T*Under this protocol,
FragMemstays within-0.11/-0.24/-0.16pp ofClass-identity oracle (*)and leads current-batchNoMem-RidgeCV(54.46/82.00/56.74%, gap+8.86/+5.29/+7.46pp):FedProtodoes not resolve fragment-recurrent drift. Averaging a single global centroid per label index c collapses distinct conditional states sc when class-to-label mappings permute across clients (FedProto-GlobalProto:20.28 / 23.24 / 25.11%). Even after development-tuned adaptation,FedProto-LocalHeadtrailsFragMem(T*)by+6.89,+4.83, and+7.13pp across all 5 seeds, whereasFragMempreserves(label c, slot j)states and splices rows directly.FedDG-ELCFS,FedTHE,FedICON) target shifts in Pt(X). Under fixed P(X) and unknown label permutation Pt(Y∣X) without labelled cues, unlabelled batches share identical feature marginals, BatchNorm/normalization statistics, and prediction entropy across permutations—leaving label remapping unidentifiable from unlabelled inputs alone. Applying Tent-style linear-head entropy minimization on Deval (FedAvg-LinEntropy, a lightweight linear probe rather than deep BatchNorm tuning) shiftsFedAvgby only+0.01,0.00, and+0.07pp (GTSRBdev pickedT0,0steps;22.91 / 30.68 / 24.89%), while tunedFedAvg(T*)trailsFragMem(T0)by+6.69,+4.42, and+5.40pp (5/5 seeds). We cite and formalize these boundaries in Section 2 (p. 3), Section 5.2 (Table 3a, p. 8), and Appendix I (Table 9, p. 26).