When Fragments Recur: Compositional Recall under Distributed Concept Drift

In long-running federated systems, clients may revise different parts of their class-to-label mappings at different times, for example when hospitals adopt updated coding conventions for different disease groups or moderation systems update policies for different content categories. A client can then face a complete labelling function that has never appeared before, although each of its local components has already been learned by other clients or in earlier rounds. Existing methods for recurring concept drift store and retrieve complete models, so this distributed historical knowledge cannot be recombined. We call this setting fragment-recurrent concept drift. On constructed CIFAR-100, GTSRB and 20 Newsgroups benchmarks with frozen encoders and current labels available before prediction, whole-head memory trails a memory that assembles rows by class state by 23–32 percentage points at switches to unseen combinations, even with true identities given. We propose FragMem, a server-side compositional memory with historical state slots shared across clients and rounds: current class-conditional feature statistics select one slot per label, the selected classifier rows form the client's model, and the server aggregates locally trained updates by label and slot, so clients with different concepts improve shared class-level knowledge. FragMem comes within 0.3 points of the class-state oracle; its advantage over whole-concept memory grows with the demand for unseen combinations and shrinks when states are hard to identify or current-batch adaptation suffices. The results support persistent federated memory that recombines knowledge from recurring concept fragments.