acceptodds
Under review as a conference paper at ICLR 2027

When Fragments Recur: Compositional Recall under Distributed Concept Drift

Abstract

In long-running federated systems, clients may revise different parts of their class-to-label mappings at different times, for example when hospitals adopt updated coding conventions for different disease groups or moderation systems update policies for different content categories. A client can then face a complete labelling function that has never appeared before, although each of its local components has already been learned by other clients or in earlier rounds. Existing methods for recurring concept drift store and retrieve complete models, so this distributed historical knowledge cannot be recombined. We call this setting fragment-recurrent concept drift. On constructed CIFAR-100, GTSRB and 20 Newsgroups benchmarks with frozen encoders and current labels available before prediction, whole-head memory trails a memory that assembles rows by class state by 23–32 percentage points at switches to unseen combinations, even with true identities given. We propose FragMem, a server-side compositional memory with historical state slots shared across clients and rounds: current class-conditional feature statistics select one slot per label, the selected classifier rows form the client's model, and the server aggregates locally trained updates by label and slot, so clients with different concepts improve shared class-level knowledge. FragMem comes within 0.3 points of the class-state oracle; its advantage over whole-concept memory grows with the demand for unseen combinations and shrinks when states are hard to identify or current-batch adaptation suffices. The results support persistent federated memory that recombines knowledge from recurring concept fragments.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
federated learning, concept drift, recurring concept drift, federated memory, compositional recall, distribution shift

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EJqsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

The authors address a practical issue in federated learning known as "fragmentary recurring concept drift": the complete label mapping currently held by a client may be novel, yet the constituent local mapping rules have already been learned by other clients or in previous rounds. To tackle this, the authors propose FragMem, a method where the server maintains a persistent "label-state" memory bank. Clients use feature statistics from their current labeled batches to match historical states, retrieving classifier rows for specific labels to assemble a classification head; following local training, the server aggregates updates based on label and state slots, enabling the reuse of local knowledge across clients and time. Evaluations on three constructed drift benchmarks show that FragMem performs close to an oracle that knows the true label states and significantly outperforms a baseline using whole-model memory. Extended experiments further analyze the benefits and failure boundaries under conditions such as many-to-one mapping, label sparsity, class imbalance, real-world sentiment grouping, and strong feature representations.

Critical strengths
  1. The authors propose two key concepts in federated learning—"reproduction of complete concepts" and "formation of new combinations from known fragments"—and point out that the storage granularity of historical knowledge limits its reusability. This offers a valuable research perspective on federated concept drift.

  2. The experimental analysis is comprehensive, covering aspects such as H1 negative controls, changes in recombination requirements, and recovery curves following the switch. Additionally, the study reports experiments addressing label sparsity and imbalance and clearly delineates the boundaries between theoretical assumptions and statistical analysis, validating the method's robustness.

Critical weaknesses
  1. The concept proposed by the authors appears quite similar to TTA for FedDG and FedProto; I believe the authors should compare their approach with these methods in the experiments and clarify the boundaries and distinctions in the "Related Work" section.

2.The authors did not conduct tests on large-scale ViT models; given that current models are predominantly Transformer-based, thorough validation on this architecture is required.

Questions for authors
  1. I believe the framework diagram in the paper needs to be redrawn; although I have provisionally given a positive rating, I hope the authors will treat the submission seriously. I will lower the score if the author does not address my questions.
  2. Refer to the issues mentioned in the weakness.
6 comments

Author Response (1/3)

We thank the reviewer for the positive rating, for recognizing our fragment-versus-whole-concept storage distinction, and for clear guidance on scope and presentation. Across three comments—integrated into our companion revised manuscript with new and substantively revised passages, figures, and tables highlighted in red (plus a full latexdiff PDF and a completely redrawn Figure 2 on page 5)—we address (1) FedProto and federated domain generalization (FedDG) / test-time adaptation (TTA) (W1, Q2), (2) DINOv2 ViT-S/B/L validation (W2), and (3) the redrawn Figure 2 (Q1).


Experimental comparisons and conceptual boundaries relative to FedProto and TTA / FedDG

W1: The concept proposed by the authors appears quite similar to TTA for FedDG and FedProto; I believe the authors should compare their approach with these methods in the experiments and clarify the boundaries and distinctions in the "Related Work" section.

Q2: Refer to the issues mentioned in the weakness.

Thank you for highlighting these comparisons. Table 6 in the initial submission (Table 7, Appendix H, p. 23 of the revised manuscript) already reports unadapted FedProto on H2 (38.4 / 53.8 / 36.4% local model and 20.3 / 23.2 / 25.1% global prototype inference vs. 63.1 / 86.6 / 64.4% for FragMem). To compare fine-tuned FedProto variants and a Tent-style linear-head entropy minimization probe (FedAvg-LinEntropy) under matched round-tt access (20 clients, T=100T=100 rounds including a 20-round burn-in on fixed 1,920/480 splits; H2: unseen recombinations; T0: 0 steps; best tier T* locked on seeds 42–43 and confirmed on seeds 44–48), each method tunes a disposable head copy on Dtraink,t\mathcal{D}_{\text{train}}^{k,t} (00, 55/2020 full-batch steps, or 11/22/55 50-sample mini-batch epochs) before predicting Devalk,t\mathcal{D}_{\text{eval}}^{k,t} (FedAvg-LinEntropy instead minimizes (W,b)(W,b) prediction entropy directly on unlabelled Devalk,t\mathcal{D}_{\text{eval}}^{k,t}, with zero persistent writeback).

Method (H2 switch %)CIFAR-100 (T0/T*)GTSRB (T0/T*)20NG (T0/T*)FragMem - Method @ T*
FragMem63.11/63.3286.61/87.2963.97/64.20—
FedProto-LocalHead38.36/56.4353.79/82.4736.44/57.07+6.89/+4.83/+7.13
FedProto-LocalProtoBlend49.58/49.5858.75/58.7556.91/56.91+13.74/+28.54/+7.29
FedProto-GlobalProto20.28/20.2823.24/23.2425.11/25.11+43.04/+64.05/+39.09
FedAvg-LinEntropy (Tent-style)22.90/22.9130.68/30.6824.81/24.89+40.41/+56.61/+39.32
FedAvg22.90/56.4230.68/82.1924.81/58.57+6.90/+5.10/+5.63

Under this protocol, FragMem stays within -0.11/-0.24/-0.16 pp of Class-identity oracle (*) and leads current-batch NoMem-RidgeCV (54.46/82.00/56.74%, gap +8.86/+5.29/+7.46 pp):

  1. Why FedProto does not resolve fragment-recurrent drift. Averaging a single global centroid per label index cc collapses distinct conditional states scs_c when class-to-label mappings permute across clients (FedProto-GlobalProto: 20.28 / 23.24 / 25.11%). Even after development-tuned adaptation, FedProto-LocalHead trails FragMem(T*) by +6.89, +4.83, and +7.13 pp across all 5 seeds, whereas FragMem preserves (label c, slot j) states and splices rows directly.
  2. Why unlabelled TTA / FedDG fails under pure concept drift. Covariate-shift TTA and FedDG (FedDG-ELCFS, FedTHE, FedICON) target shifts in Pt(X)P_t(X). Under fixed P(X)P(X) and unknown label permutation Pt(Y∣X)P_t(Y\mid X) without labelled cues, unlabelled batches share identical feature marginals, BatchNorm/normalization statistics, and prediction entropy across permutations—leaving label remapping unidentifiable from unlabelled inputs alone. Applying Tent-style linear-head entropy minimization on Deval\mathcal{D}_{\text{eval}} (FedAvg-LinEntropy, a lightweight linear probe rather than deep BatchNorm tuning) shifts FedAvg by only +0.01, 0.00, and +0.07 pp (GTSRB dev picked T0, 0 steps; 22.91 / 30.68 / 24.89%), while tuned FedAvg(T*) trails FragMem(T0) by +6.69, +4.42, and +5.40 pp (5/5 seeds). We cite and formalize these boundaries in Section 2 (p. 3), Section 5.2 (Table 3a, p. 8), and Appendix I (Table 9, p. 26).

Author Response (2/3)

Validation on large-scale Vision Transformers (DINOv2 ViT-S/14, ViT-B/14, and ViT-L/14) and full 50k/10k CIFAR-100

W2: The authors did not conduct tests on large-scale ViT models; given that current models are predominantly Transformer-based, thorough validation on this architecture is required.

Thank you for pointing out the need for prominent Vision Transformer validation—our initial submission placed DINOv2 in an appendix table and omitted larger ViT backbones. First, our submitted Table 4b already evaluated frozen DINOv2-small (ViT-S/14) on CIFAR-100 H2: with an MLP head, FragMem reaches 84.2% (matching Class-identity oracle (*)), leading fine-tuned FedAvg+FT (80.9%) by +3.23 pp, FedBABU (76.4%) by +7.7 pp, FEROMA-τ (65.6%), and Concept-unit control (62.7%), and reaches 83.6% with a linear head (95% CIs in main-text Table 3b, Section 5.2, p. 8).

Second, to validate larger Transformer encoders at full dataset scale with strictly zero test leakage, we completed new experiments on full 50,000/10,000 CIFAR-100 using frozen DINOv2 ViT-B/14 (both C=20C=20 superclasses and C=100C=100 fine classes with B=20B=20 blocks and batch size 500) and DINOv2 ViT-L/14 (C=20C=20), projected with strictly inductive train-only PCA-128 fit on outer training splits alone (hyperparameters locked on 80/20 training-split holdouts across seeds 42–43, confirmed on seeds 44–48):

Full 50k/10k Pool (H2, T0/T*)FragMemBestWhole (oracle wr.)Concept-unit ctrl (best-loss)FedAvgFedProto-LocalHeadClass-identity oracle (*)
DINOv2 ViT-B/14 (C=20C=20)91.91/91.9671.39/89.0971.35/89.0535.51/88.6469.86/88.4491.93/91.98
DINOv2 ViT-L/14 (C=20C=20)94.53/94.5772.65/92.0972.62/92.0836.63/91.8771.21/91.6994.53/94.57
DINOv2 ViT-B/14 (C=100C=100, B=20B=20)85.75/86.3377.79/84.0277.06/83.3931.08/82.2975.15/81.8986.02/86.57

Across all three large-scale ViT benchmarks, FragMem delivers consistent gains across the 5 confirmation seeds even when stronger Transformer features raise overall accuracy ( in Table 12, p. 29):

  • Consistent margins over whole-head and fine-tuned baselines. At development-selected T∗T^*, FragMem leads BestWhole (oracle write) by +2.87 / +2.48 / +2.31 pp and FedAvg by +3.32 / +2.70 / +4.04 pp across the three pools. Even unadapted FragMem(T0) leads fine-tuned BestWhole(T*) by +2.82 / +2.44 / +1.73 pp (full NoMem-RidgeCV and LossMatch scores appear in Table 12).
  • No negative transfer on H1 and 89.96x bandwidth savings at C=100C=100. On the ViT-B/14 H1 seen-concept control, the clean comparison at unadapted T0 is +0.39 pp (dev-selected T* slightly lowers Concept-unit control from 91.58% to 91.06%, widening T* to +0.92 pp). At C=100C=100 fine classes, transmitting whole-head candidate pools consumes 37,277.65 MB compared to 414.40 MB for FragMem (89.96x savings).

We placed Table 3b in Section 5.2 (p. 8) and added these full-data ViT-B/14 and ViT-L/14 results to Section 5.7 (p. 10) and Appendix I (Table 12, p. 29) of the revised manuscript.

Author Response (3/3)

Redrawing Figure 2 into a three-lane, five-step protocol diagram

Q1: I believe the framework diagram in the paper needs to be redrawn; although I have provisionally given a positive rating, I hope the authors will treat the submission seriously. I will lower the score if the author does not address my questions.

Thank you for raising this directly—we take your feedback very seriously. Looking back at the submitted Figure 2, we agree that packing client updates, server lookups, and round timing into one dense diagram obscured both data isolation and execution order.

We have therefore completely redrawn Figure 2 in the revised manuscript (Section 4, p. 5, with caption revisions highlighted in red) as a three-lane vector diagram (Client k, Server, and Persistent Slot Bank) that separates each round tt into five explicit steps synchronized with Section 4 notation:

  1. Step 1 — Batch Isolation (obs. c vs. state s_c). The Client k lane explicitly separates frozen representation ϕ\phi over the disjoint round-tt training batch Dtraink,t\mathcal{D}_{\text{train}}^{k,t} (with observed labels obs. c mapped from latent state state s_c) and held-out evaluation batch Devalk,t\mathcal{D}_{\text{eval}}^{k,t}.
  2. Step 2 — Moment Upload. From ϕ(Dtraink,t)\phi(\mathcal{D}_{\text{train}}^{k,t}) alone, the client computes and uploads per-label counts ncn_c, means mcm_c, within-label second moments SS\mathrm{SS}, and split-half statistic zz for calibrating novelty threshold τ\tau.
  3. Step 3 — Slot Match / Spawn & Assembly. Without any oracle state IDs, Eq. (1) matches each observed label cc against stored descriptors and rows (ξc,j,rc,j)(\xi_{c,j}, r_{c,j}) in the persistent (label, slot) bank, selects or spawns active slot index ak,cta^t_{k,c}, and splices the assembled head Θkt=Stack⁡crc,ak,ct\Theta_k^t = \operatorname{Stack}_c r_{c,a^t_{k,c}}.
  4. Step 4a — Pre-Train Serving. The assembled model Θkt\Theta_k^t (strictly from historical writes ≤t−1\le t{-}1) predicts Devalk,t\mathcal{D}_{\text{eval}}^{k,t} first, visually separating evaluation from local training; the optional client-side +FT control on Dtraink,t\mathcal{D}_{\text{train}}^{k,t} is drawn on a dotted auxiliary clone branch inside the Client k lane.
  5. Steps 4b–5 — Persistent Writeback. Only after evaluation do clients run persistent 10-epoch SGD on Dtraink,t\mathcal{D}_{\text{train}}^{k,t} and upload updated classifier rows and feature moments, which the server aggregates strictly by (c, j) slot key back into the persistent bank for round t+1t+1.

The diagram in Section 4 (p. 5) and its walkthrough in Appendix C (p. 17, which groups Steps 1–2 under its first bullet) now map step-by-step in our revised PDF.

Reviewer_mWtYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the problem of recurring class-to-label fragments in federated learning, where fragments observed earlier reappear in new combinations. The proposed framework, FragMem, uses feature statistics to match current label states to historical ones, compiles classifier rows, and combines updates across clients and rounds. In controlled experiments, the method shows accuracy gains that depend on the accuracy of state identification and on how well the current-batch adaptation works.

Critical strengths
  1. The method is clearly explained and sound. The recall-set separation explains why a stored whole head cannot directly retrieve unseen combinations. In contrast, the persistent label-state memory provides a way to share knowledge across clients and rounds. The paper provides useful and informative comparisons showing the benefit of storing and reusing parts of a concept. The experiments explore different ways of identifying states and organising memory, while also testing different levels of recombination and class imbalance.

  2. Experiments. Different ways of identifying states and storing past knowledge are compared, showing the benefits of reusing parts of a concept. The authors also test different patterns of recurrence, class imbalance, label mappings and feature representations. The experiments show where FragMem performs well and where its gains are smaller than those of other methods.

Critical weaknesses
  1. The identification assumptions need further testing. FragMem assumes that the class-conditional feature distributions are shared across clients. In particular, the experiments use a common feature pool, which means the authors do not test what happens when these distributions differ between clients. This is an important point that would be worth exploring and discussing. For example, if the distributions differ between clients, this could lead to incorrect state matches.

  2. The sources of the improvement are not well separated. From the current experiments, it is not clear how much FragMem benefits from combining learned parts in new ways, and how much from accumulating training updates for the same stored parts across different concepts. It would be useful to separate these two effects and show how much each contributes to the accuracy improvement.

Questions for authors
  1. Differences between clients: Could the authors test FragMem when clients use the same label mappings but have different feature distributions for the same classes? How does this affect state matching, the number of memory slots created, and accuracy?

  2. Sources of the accuracy gains: Could the authors clarify how much FragMem gains from combining learned parts in new ways, and how much from accumulating updates for the same parts across concepts? Would a comparison with a similar amount of training for each stored part help separate these contributions?

3 comments

Author Response (1/2)

We thank the reviewer for the positive evaluation and two sharp mechanistic questions. Across two comments—backed by new experiments added in red to Section 5.4, Section 5.7, and Appendix I of our revised manuscript—we address cross-client feature shifts (W1, Q1) here and compositional assembly versus update accumulation (W2, Q2) in Comment (2/2).


Cross-client feature distribution shifts, state matching, and client batch centering

W1: The identification assumptions need further testing. FragMem assumes that the class-conditional feature distributions are shared across clients. In particular, the experiments use a common feature pool, which means the authors do not test what happens when these distributions differ between clients. This is an important point that would be worth exploring and discussing. For example, if the distributions differ between clients, this could lead to incorrect state matches.

Q1: Differences between clients: Could the authors test FragMem when clients use the same label mappings but have different feature distributions for the same classes? How does this affect state matching, the number of memory slots created, and accuracy?

Thank you for raising this question. Across confirmation seeds 44–48 on the three 20-class 128-d splits (20 clients, T=100T=100 rounds after a 20-round burn-in; 400 train and 400 held-out eval samples/round), we evaluated STATIC (fixed concept schedule without drift; T0 round-average test accuracy) and H2 (unseen recombinations; T0 switch accuracy at 0 adaptation steps). none, submode tilt (0.5) (η=0.5\eta=0.5 two-submode sampling tilt fit on train), and rigid client offsets (0.5) (X′=X+0.5 s vkX'=X+0.5\,s\,v_k, train-only radius ss) use a common-random-numbers inverse-CDF sampler per seed—so this suite's none reference (63.23/86.26/64.01%) differs slightly from Table 9's sample-drawn runs. FragMem-ClientCenter subtracts Dtraink,t\mathcal{D}_{\text{train}}^{k,t}'s batch mean from train and eval, canceling any constant translation 0.5 s vk0.5\,s\,v_k to numerical precision by construction (rigid / ClientCenter equals none / ClientCenter as a translation diagnostic).

Shift & Method (C100 / GTSRB / 20NG, T0)H2 switch %H2 slots / stateSTATIC test %STATIC slots / state
none / FragMem63.23/86.26/64.011.34/1.23/1.6664.32/88.57/64.832.06/1.61/2.40
none / ClientCenter63.20/86.16/63.611.19/1.11/1.4964.35/88.45/64.641.54/1.39/2.10
submode 0.5 / FragMem63.86/86.02/63.992.27/2.10/2.3365.37/88.32/64.942.66/2.30/3.35
rigid 0.5 / FragMem55.34/77.04/53.5416.42/15.95/17.1264.34/88.22/63.9221.49/21.27/24.49
rigid 0.5 / ClientCenter63.20/86.16/63.611.19/1.11/1.4964.35/88.45/64.641.54/1.39/2.10
rigid 0.5 / Class-identity oracle (*)62.34/85.47/62.03—63.52/87.15/62.57—

Under fixed 5-epoch +FT, FragMem scores 64.35/86.95/64.14% on submode 0.5 versus 58.70/82.99/58.61% for Concept-unit control (best-loss), leading by +5.65 / +3.96 / +5.53 pp (full +FT rows in Table 11, p. 28). Three findings stand out (Section 5.7, p. 10; Table 11, p. 28, with 95% CIs):

  1. Submode tilt (0.5) is benign. FragMem (T0) changes by +0.63, -0.23, and -0.02 pp (2.10–2.33 slots/state on H2; 2.30–3.35 on STATIC).
  2. Rigid offsets (0.5) trigger slot splitting masked by purity. Constant per-client shifts fragment states into 15.95–17.12 (H2) and 21.27–24.49 (STATIC) client-specific slots, dropping FragMem H2 accuracy by -7.89 / -9.22 / -10.47 pp (Class-identity oracle: -2.04 to -0.95 pp). Yet slot purity remains 0.989–1.000 because each slot holds a single client, showing that slots/state captures fragmentation missed by purity.
  3. Batch centering cancels constant translations. Subtracting Dtraink,t\mathcal{D}_{\text{train}}^{k,t}'s mean collapses bloat to 1.11–1.49 (H2) and 1.39–2.10 (STATIC) slots/state and recovers +7.87 / +9.12 / +10.08 pp on H2, within 0.02–0.51 pp of centered Class-identity oracle (*).

Author Response (2/2)

Disentangling compositional row assembly from cross-concept update accumulation

W2: The sources of the improvement are not well separated. From the current experiments, it is not clear how much FragMem benefits from combining learned parts in new ways, and how much from accumulating training updates for the same stored parts across different concepts. It would be useful to separate these two effects and show how much each contributes to the accuracy improvement.

Q2: Sources of the accuracy gains: Could the authors clarify how much FragMem gains from combining learned parts in new ways, and how much from accumulating updates for the same parts across concepts? Would a comparison with a similar amount of training for each stored part help separate these contributions?

Thank you for encouraging us to separate compositional assembly from accumulated per-part training. Because online runs simultaneously evolve bank slots and update counts, we designed a controlled frozen-bank diagnostic across CIFAR-100, GTSRB, and 20 Newsgroups (Section 5.4, pp. 8–9 and Appendix I, Table 10, p. 27). During burn-in rounds 0–19, both comparison banks use oracle identity writes under equal total full-batch SGD steps across e=1 (1 persistent full-batch epoch/round, 400 total steps) or e=10 (10 epochs/round, 4,000 total steps), and are then completely frozen in rounds 20–99 (zero persistent writes).

Specifically, we freeze two burn-in banks: a class-slot bank (CP), which pools per-label updates across concepts sharing a class state, and a whole-concept bank (WT), which isolates heads per complete concept. In rounds 20–99, we compare reading a whole head from one stored concept against splicing individual classifier rows across concepts via oracle identities (OracleID / OracleOverlap) or empirical matchers (Fingerprint / LossMatch / BestLoss). Keeping total burn-in optimizer steps equal cleanly isolates row recombination within the same WT bank from cross-concept update pooling (CP - WT).

Frozen-Bank H2 Unseen-Combination Contrast (C100 / GTSRB / 20NG, pp)e=1 (400 steps)e=10 (4,000 steps)
H1 seen control: CP-Compose-OracleID - CP-Whole-OracleOverlap—0.00/0.00/0.00
Within-bank CP (oracle): CP-Compose-OracleID - CP-Whole-OracleOverlap+15.89/+19.60/+21.79+21.28/+28.56/+23.30
Within-bank CP (empirical read): CP-Compose-Fingerprint - CP-Whole-BestLoss+15.35/+19.12/+21.27+21.47/+28.98/+23.10
Within-bank WT (oracle): WT-RowSplice-OracleID - WT-Whole-OracleOverlap+16.05/+18.65/+24.22+22.79/+29.95/+27.26
Within-bank WT (LossMatch): WT-RowSplice-LossMatch - WT-Whole-BestLoss+16.47/+19.47/+24.36+23.35/+32.39/+25.24
Between banks (CP vs WT): CP-Compose-OracleID - WT-RowSplice-OracleID+2.73/+5.22/+2.49+3.70/+5.86/+0.98

Two takeaways emerge clearly from this decomposition:

  1. Within-bank compositional row assembly drives most of the gain. Splicing existing rows within the fixed WT bank alone improves unseen-combination accuracy at e=10 by 22.79 to 29.95 pp (OracleID) and 23.35 to 32.39 pp (LossMatch), exceeding CP's like-for-like oracle gain (21.28 to 28.56 pp; empirical CP-Compose-Fingerprint adds +21.47 / +28.98 / +23.10 pp on H2 vs. 0.00/0.00/-0.01 on H1). On unpooled WT banks, WT-RowSplice-Fingerprint (45.06–59.77%) trails LossMatch, consistent with smaller per-concept slot sample counts making moment matching noisier.
  2. Estimated residual CP - WT margins are smaller and uncertain. Cross-concept pooling adds only +3.70 / +5.86 / +0.98 pp at e=10 (+2.49 to +5.22 pp at e=1; Table 10, p. 27), with 20NG's e=10 interval (+0.98 [-1.58, +3.54]) and two of three e=1 intervals crossing zero. Residual CP - WT differences co-occur with unequal per-row update counts (smaller row-norm CV, SD/mean: 0.189–0.251 in CP vs. 0.220–0.287 in WT at e=10) and joint softmax competition. Thus, compositional row assembly within frozen banks exceeds cross-concept pooling.

Many thanks for all the extra experiments and for clarifying my questions. The new takeaways definitely add to the paper and provide useful insights.

Reviewer_VWbNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies "fragment-recurrent" concept drift in FL, where each client's class-to-label mapping is a combination of per-block fragments, so a client can land on a full mapping nobody has seen even though every piece was seen before. FragMem keeps a server-side bank of slots per label (feature-statistics fingerprint plus a classifier row), matches each label of the current labelled batch to a slot with a chi-square style test, builds the head from the picked rows and averages updates per (label, slot). On frozen-feature pools from CIFAR-100, GTSRB and 20 Newsgroups it is within 0.3 points of a class-identity oracle and 23-32 points above a concept-identity reference.

Critical strengths
  1. The setting is clearly defined and the "known parts, new whole" idea is easy to follow.
  2. The paper is unusually honest about its limits. Negative controls (H1), dose-response curves, failure modes (lump pool, imbalance, few labels) and an audit of which baselines read the batch before serving are all reported.
  3. The method is simple and cheap (about 2x FedAvg bytes), and the novelty threshold is calibrated without true state labels.
Critical weaknesses
  1. The headline 23-32 point gap is against a weak reference. At an unseen concept it just copies the client's previous head, which is stale on the changed block by construction (the authors admit it isn't the best stored head). The fairer number is the no-memory refit on the current labelled batch, which trails FragMem by only 4.4-8.3 points on H2, and with DINOv2 features plain FedAvg fine-tuned on the batch is 3.2 points behind. So the actual value of the memory is much smaller than the abstract suggests.

  2. Most existing baselines are not run under the same protocol. FragMem, IFCA and FedRC see the labelled round-t batch before serving, while FedCCFA, FedDrift, FLUX etc. serve a model from round t-1. Since the main metric is switch-round accuracy, this alone explains a lot of the 16-40% vs 63% gap in Table 1. I'd want every baseline given the same batch (at least fine-tuned on it before predicting) on the three main pools, not only on DINOv2.

  3. The benchmark is very small and synthetic. Each pool is 20 classes with 120 samples per class, so every client-round resamples from the same 1,920 train and 480 eval examples, with a linear head on 128-d PCA features (PCA also fit on the eval inputs). Drift is label permutations in blocks of 4 with K=6 and B=5, so H2 basically never repeats a whole concept and the setup is built to favor class-level memory. Part of the gap is also just data exposure, since coverage explains only 10.7 of the 23.3 points.

  4. When things get more realistic the gains mostly go away. On GoEmotions, the only real label convention, FragMem beats concept units by just 0.62 on H2, loses on H1, and purity is about 0.67. Under Dirichlet 0.5 it is 3.9-7.0 points below the concept reference on H1. And the hospital-coding motivation is really about new or split codes, not permutations of a fixed label set, which isn't tested.

  5. Novelty is moderate. A per-label slot bank with a mean-difference test is close to FedCCFA's classifier clustering plus a persistent bank, and the fingerprint follows RKME/Model Spider. Prop. 1 is a counting argument (the proof is "immediate") and the accuracy form assumes stale rows get exactly 0 accuracy. Work on mixing models across clusters or clustering from data statistics is missing, e.g. FedSoft (Ruan and Joe-Wong, 2022), FedPAC (Xu et al., 2023) and PACFL (Vahidian et al., 2023). The privacy cost of uploading per-label means and second moments every round isn't discussed either.

  6. The claim that the memory unit, not the matcher, is what matters doesn't hold up beyond the easy pools. Loss matching is within 0.6-3.5 points on the main pools but 9.7 points worse on the lump pool and about 12 worse on GoEmotions. Also the writing is very dense, almost every sentence packs several numbers and hedges, and some controls are confusing ("FragMem, concept units" also changes three other settings, the "earlier port (audited)" rows in Table 6).

Questions for authors
  1. How do FedCCFA, FLUX, FedDrift and FedAvg compare when fine-tuned on the round-t labelled batch before serving, on the ResNet-18 pools?
  2. What happens with a concept reference that picks the best stored head for the batch (and maybe fine-tunes it) instead of copying the previous one?
  3. Can you show results on a larger pool (more classes, full data) or with a trainable encoder?
  4. How would FragMem handle label-space growth or splits, like new diagnosis codes?
5 comments

Author Response (1/5)

We thank the reviewer for the constructive critique and for recognizing our setting formulation and boundary reporting. Across five comments—paired with a revised manuscript with new and substantively revised passages, figures, and tables highlighted in red (plus a full latexdiff PDF)—we address (1) adapted refits and best-stored-head baselines (W1, Q2), (2) symmetric pre-prediction fine-tuning (W2, Q1), (3) strictly inductive train-only PCA, full-dataset ViT/C=100 scaling, and exposure controls (W3, Q3), (4) challenging drift regimes, dynamic label spaces, and matcher/prose clarity (W4, Q4, W6), and (5) related-work taxonomy, Proposition 1, and privacy (W5).


Recalibrating the 23–32 point reference against adapted refits and best-stored-head baselines

W1: The headline 23-32 point gap is against a weak reference. At an unseen concept it just copies the client's previous head, which is stale on the changed block by construction (the authors admit it isn't the best stored head). The fairer number is the no-memory refit on the current labelled batch, which trails FragMem by only 4.4-8.3 points on H2, and with DINOv2 features plain FedAvg fine-tuned on the batch is 3.2 points behind. So the actual value of the memory is much smaller than the abstract suggests.

Q2: What happens with a concept reference that picks the best stored head for the batch (and maybe fine-tunes it) instead of copying the previous one?

Thank you for raising this central fairness point. We agree that the 23–32 percentage-point margin against the unadapted concept-identity reference (Table 1, copying the client's previous head) isolates storage granularity rather than single-round local adaptation. Our initial submission already noted that single-round no-memory refits trail FragMem by 8.25 / 4.44 / 7.49 pp (Section 5.3) and fine-tuned DINOv2-small FedAvg+FT trails by 3.23 pp (now in Table 3b, Section 5.2, p. 8; formerly Appendix Table 4b). In the revised manuscript, we pair these adapted margins directly alongside the 23–32 point reference in the Abstract and Introduction (highlighted in red).

To answer Q2 under matched current-batch access (20 clients, T=100T=100 rounds including a 20-round burn-in on fixed 1,920/480 20-class splits; 400 disjoint labelled train and 400 unlabelled eval samples/round; H1: seen concepts, H2: unseen recombinations), each learner tunes a disposable copy of its served head on Dtraink,t\mathcal{D}_{\text{train}}^{k,t} before predicting Devalk,t\mathcal{D}_{\text{eval}}^{k,t} (T0=0T_0 = 0, 55/2020 full-batch steps, or 11/22/55 50-sample mini-batch epochs; best tier T∗T^* locked on dev seeds 42–43 and confirmed on seeds 44–48, with zero persistent writeback). Here the Class-identity oracle (*) matches Table 1, the Concept-unit control (best-loss) picks the stored whole head with lowest training loss, and BestWhole (oracle write) uses true concept IDs for persistent writes while selecting at serving time by lowest cross-entropy on Dtraink,t\mathcal{D}_{\text{train}}^{k,t}.

Reference Method (H2 switch %)CIFAR-100 (T0/T*)GTSRB (T0/T*)20NG (T0/T*)FragMem - Method @ T*
FragMem63.11/63.3286.61/87.2963.97/64.20—
Class-identity oracle (*)63.23/63.4386.90/87.5364.16/64.36-0.11/-0.24/-0.16
BestWhole (oracle write)41.67/57.0256.73/82.5741.59/58.54+6.30/+4.73/+5.66
Concept-unit control (best-loss)41.57/57.0056.64/82.5741.54/58.46+6.32/+4.72/+5.74
Concept-identity reference (*)39.79/56.9554.88/82.7038.40/58.16+6.37/+4.60/+6.04
NoMem-RidgeCV (T0)54.46/54.4682.00/82.0056.74/56.74+8.86/+5.29/+7.46

Replacing the previous-head concept-identity reference with best-stored-head retrieval (BestWhole) still cannot recover unseen combinations at T0T_0. Even after development-tuned fine-tuning (+FT), FragMem(T*) leads BestWhole (oracle write) by +6.30 / +4.73 / +5.66 pp across all 5 confirmation seeds (95% tt (df=4) CIs in Section 5.2, p. 8 and Table 9, p. 26) while staying within 0.11–0.24 pp of the Class-identity oracle (*). On CIFAR-100, transmitting candidate whole heads for the concept-unit control also requires 1,505.68 MB (18.17x FragMem's 82.88 MB; Table 9, p. 26).

Author Response (2/5)

Symmetric pre-prediction fine-tuning across FedCCFA, FLUX, FedDrift, FedAvg, and per-class loss matching

W2: Most existing baselines are not run under the same protocol. FragMem, IFCA and FedRC see the labelled round-t batch before serving, while FedCCFA, FedDrift, FLUX etc. serve a model from round t-1. Since the main metric is switch-round accuracy, this alone explains a lot of the 16-40% vs 63% gap in Table 1. I'd want every baseline given the same batch (at least fine-tuned on it before predicting) on the three main pools, not only on DINOv2.

Q1: How do FedCCFA, FLUX, FedDrift and FedAvg compare when fine-tuned on the round-t labelled batch before serving, on the ResNet-18 pools?

Thank you for requesting a symmetric evaluation of round-(t−1)(t-1) baselines. Under the matched pre-prediction +FT protocol in Comment (1/5) (20 clients, T=100T=100 rounds, tiers T∗T^* locked on dev seeds 42–43 and confirmed on seeds 44–48), every baseline adapts a disposable copy of its served classifier head on Dtraink,t\mathcal{D}_{\text{train}}^{k,t} over the same step/epoch grid before predicting Devalk,t\mathcal{D}_{\text{eval}}^{k,t}, with zero persistent writeback.

Method (H2 switch %)CIFAR-100 (T0/T*)GTSRB (T0/T*)20NG (T0/T*)FragMem - Method @ T*
FragMem63.11/63.3286.61/87.2963.97/64.20—
Per-class loss matching62.67/62.9883.78/85.9063.80/64.11+0.34/+1.39/+0.09
Concept-unit control (best-loss)41.57/57.0056.64/82.5741.54/58.46+6.32/+4.72/+5.74
BestWhole (oracle write)41.67/57.0256.73/82.5741.59/58.54+6.30/+4.73/+5.66
FedCCFA40.78/56.7556.38/82.5539.07/58.23+6.57/+4.74/+5.97
FLUX39.89/56.9955.03/82.7138.42/58.13+6.33/+4.59/+6.08
FedDrift33.78/56.5844.31/82.3729.82/58.08+6.74/+4.92/+6.12
FedAvg22.90/56.4230.68/82.1924.81/58.57+6.90/+5.10/+5.63
NoMem-RidgeCV (T0)54.46/54.4682.00/82.0056.74/56.74+8.86/+5.29/+7.46

This unified pre-prediction comparison yields two clear findings across all 5 confirmation seeds:

  1. Whole-head baselines hit a common adaptation plateau below compositional recall. Under development-selected pre-prediction fine-tuning (100k–120k total gradient steps versus 30k–60k for FragMem), all six displayed whole-head baselines (seven including the concept-identity reference in Comment 1/5) fall into a tight band (56.42–57.02%, 82.19–82.71%, 58.08–58.57%), only slightly above the closed-form current-batch ridge solution NoMem-RidgeCV (54.46 / 82.00 / 56.74%). Single-batch adaptation alone cannot reconstruct cross-client fragment combinations from a 400-sample batch as accurately as recalling historically consolidated class rows.
  2. FragMem leads even without pre-prediction fine-tuning. Evaluated strictly at T0T_0 (20k total steps), unadapted FragMem still outperforms 5-epoch fine-tuned BestWhole (oracle write) by +6.09 / +4.04 / +5.43 pp across all 5 seeds, with similar +4.0 to +6.7 pp leads over the concept-unit control and FedAvg (Section 5.2, Table 3a, p. 8; Appendix I, Table 9, p. 26).

Author Response (3/5)

Inductive train-only PCA, full-dataset ViT and 100-class scaling, trainable encoders, and exposure attribution

W3: The benchmark is very small and synthetic. Each pool is 20 classes with 120 samples per class, so every client-round resamples from the same 1,920 train and 480 eval examples, with a linear head on 128-d PCA features (PCA also fit on the eval inputs). Drift is label permutations in blocks of 4 with K=6 and B=5, so H2 basically never repeats a whole concept and the setup is built to favor class-level memory. Part of the gap is also just data exposure, since coverage explains only 10.7 of the 23.3 points.

Q3: Can you show results on a larger pool (more classes, full data) or with a trainable encoder?

Thank you for scrutinizing benchmark scale, PCA transductivity, and data exposure. Alongside the K=6,B=5K=6, B=5 H2 benchmark, our manuscript evaluates H1 negative controls at K=6K=6 (Table 1), 48-client and 100-client federations (Tables 2b and 6; submitted Table 5), and frozen DINOv2-small features (Table 3b, Section 5.2, p. 8). We disclose explicitly that the original 1,920/480 splits fit PCA/SVD transductively on combined pool inputs, whereas our supplementary controlled runs keep exact 20 Newsgroups duplicates within one split and fit feature-norm scaling strictly on training splits (Appendix B, p. 16).

To resolve both inductive PCA and dataset/backbone scaling (W3, Q3), we added a new suite using strictly inductive train-only PCA-128 fit solely on outer training splits, locking learning rates and +FT tiers via internal 80/20 outer-train holdouts across seeds 42–43 before confirming on seeds 44–48. First, on the 2,400-sample ResNet-18 control, inductive PCA yields a +6.35 pp margin over BestWhole (matching the +6.30 pp transductive margin; 95% CIs in Table 12, p. 29), indicating that transductive PCA is not the driver. Second, across full 50,000/10,000 CIFAR-100 splits (100 classes, ViT-B/14 & ViT-L/14):

Inductive train-only PCA-128 PoolFragMem (T0/T*)BestWhole (oracle wr.)Concept-unit ctrl (best-loss)FedAvgFragMem - BestWhole @ T*
ResNet-18 (2,400, C=20C=20, H2)63.11/63.2941.47/56.9441.44/56.9422.80/56.37+6.35 [+5.85, +6.86]
ViT-B/14 (50k/10k, C=20C=20, H2)91.91/91.9671.39/89.0971.35/89.0535.51/88.64+2.87 [+2.76, +2.99]
ViT-L/14 (50k/10k, C=20C=20, H2)94.53/94.5772.65/92.0972.62/92.0836.63/91.87+2.48 [+2.33, +2.63]
ViT-B/14 (50k/10k, C=100C=100, B=20B=20, batch 500, H2)85.75/86.3377.79/84.0277.06/83.3931.08/82.29+2.31 [+2.05, +2.57]
  • Scaling and H1 control: Across full 50k/10k CIFAR-100, FragMem beats BestWhole (oracle write) by +2.31 to +2.87 pp and FedAvg by +2.70 to +4.04 pp (95% CIs in Table 12, p. 29). On the ViT-B/14 H1 seen-concept control, the fair unadapted margin at T0 is +0.39 pp (extra +FT at T* slightly lowers Concept-unit control from 91.58% to 91.06%, giving +0.92 pp). At C=100C=100, FragMem uses 414.40 MB versus 37,277.65 MB for the concept-unit control (89.96x savings).
  • Why frozen encoders fit convention drift, and compatibility with trainable backbones (Q3): Fragment-recurrent drift models changing label conventions (Pt(Y∣X)P_t(Y\mid X) shifts under stable P(X)P(X), as in revised coding or moderation rules), where freezing foundation features isolates head recombination without label-permutation interference in the backbone. Structurally, FragMem's two-level split mirrors decoupled FL (e.g., FedBABU): a shared backbone ϕ\phi can update slowly via standard FedAvg while FragMem runs fast (label, slot) head recall and consolidation on top (with exponential moving averages on moments ξc,j\xi_{c,j} to track representation drift); evaluating joint backbone training remains scoped to future work (Section 6, p. 10).
  • Frozen-bank exposure attribution (W3): Freezing oracle-written banks after rounds 0–19 (4,000 equal training steps) separates our class-slot bank (CP) from a whole-concept bank (WT). Splicing rows within WT gains +22.79 / +29.95 / +27.26 pp on H2, matching or exceeding within-CP composition gains of 21.28 to 28.98 pp across oracle/empirical reads (H1 control: 0.00 to -0.01 pp), whereas between-bank CP - WT adds only 0.98 to 5.86 pp alongside unequal per-row update counts and softmax competition. Thus, within-bank composition gains exceed residual CP - WT differences (Section 5.4, pp. 8–9; Table 10, p. 27).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank Reviewers VWbN, mWtY, and EJqs for their feedback. Reviews note three strengths: (1) the "known parts, new whole" view linking memory granularity to concept-drift reuse (VWbN, mWtY, EJqs); (2) a lightweight (label, slot) bank (~2x FedAvg bytes) with label-free threshold calibration (VWbN, mWtY); and (3) candid H1 controls, serving audits, and boundary tests (VWbN, mWtY, EJqs).

Core New Experiments

  • Pre-Prediction +FT, Best-Head Retrieval, & FedProto/TTA (VWbN W1–W2, Q1–Q2; EJqs W1, Q2): Under equal disposable tuning on Dtraink,t\mathcal{D}_{\text{train}}^{k,t} across 17 methods (BestWhole (oracle write), FedCCFA, FLUX, FedDrift, FedAvg, FedProto, Tent-style probe), whole-head/prototype baselines plateau (56.42–57.02% / 82.19–82.71% / 57.07–58.57%; entropy probe: +0.01/0.00/+0.07 pp). FragMem(T*) reaches 63.32/87.29/64.20% (0.11–0.24 pp from class oracle), beating BestWhole by +6.30/+4.73/+5.66 pp (18.17x less bandwidth) and FedProto-LocalHead by +6.89/+4.83/+7.13 pp; unadapted FragMem(T0) leads 5-epoch BestWhole by +6.09/+4.04/+5.43 pp (VWbN 1/5–2/5, EJqs 1/3; Table 3a, p. 8; Table 9, p. 26).
  • Inductive PCA, Full 50k CIFAR-100, ViT-S/B/L, & C=100C=100 (VWbN W3, Q3; EJqs W2): Train-only PCA-128 (zero test leakage) matches transductive PCA on ResNet-18 (+6.35 vs. +6.30 pp over BestWhole). On DINOv2-small (84.2%, +3.23 pp vs. FedAvg+FT, +7.7 pp vs. FedBABU) and full 50k/10k CIFAR-100 (ViT-B/14 C=20C=20, ViT-L/14 C=20C=20, ViT-B/14 C=100C=100), FragMem reaches 91.96% / 94.57% / 86.33%, beating BestWhole by +2.87/+2.48/+2.31 pp and FedAvg by +3.32/+2.70/+4.04 pp (89.96x fewer bytes at C=100C=100; VWbN 3/5, EJqs 2/3; Table 3b, p. 8; Table 12, p. 29).
  • Row Splicing vs. Pooling & Client Shifts (mWtY W1–W2, Q1–Q2; VWbN W3): Freezing equal-step (4k-step) CP/WT banks shows within-WT splicing gains +22.79/+29.95/+27.26 pp on H2 (H1: 0.00 pp), outstripping CP - WT pooling (+0.98 to +5.86 pp). Under cross-client shifts, submode tilts keep FragMem(T0) within -0.23 to +0.63 pp (+5.65/+3.96/+5.53 pp vs. whole-head control with +FT), and batch centering (FragMem-ClientCenter) cancels client offsets, restoring 1.11–1.49 slots/state within 0.02–0.51 pp of centered oracle (mWtY 1/2–2/2; Secs. 5.4, 5.7; Tables 10–11, pp. 27–28).

Revised PDF Checklist (Additions in Crimson Red)

  • Main Text (pp. 1–10): Updated Abstract/Intro (pp. 1–2); expanded FedSoft/PACFL/FedPAC, FedProto, and FedDG/TTA bounds (Sec. 2, p. 3); clarified Prop. 1 & moment privacy (Secs. 3.3, 4.1); redrew Figure 2 (p. 5) (EJqs Q1); added Table 3a–3b (p. 8), matcher scope (Sec. 5.3), frozen-bank (Sec. 5.4), and shift/ViT/C=100C=100 results (Sec. 5.7, p. 10).
  • Appendix (pp. 16–29): Added train-only notes (Apps. B–D), separated audited ports (Table 7, p. 23), and added Appendix I (pp. 25–29) with glossary and Tables 9–12 (plus latexdiff PDF).

Reading Roadmap: Point-by-point replies and tables follow under each review (VWbN 1/5–5/5, mWtY 1/2–2/2, EJqs 1/3–3/3); trainable-encoder (Q3) & dynamic-label (Q4) discussions appear in VWbN 3/5–4/5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.