Preferential Federated Learning Beyond Label Calibration

Personalized federated learning (pFL) assumes that clients differ, but usually fixes what each client keeps private before measuring how they differ. When clients are human annotators, disagreement is often treated as noise or as a difference in strictness that a per-client offset and scale can absorb. We isolate a component that no such calibration can capture: preferential concept shift, in which clients differ in their preferences, that is, in how they weigh the properties of an item, so that they label the same items but order some of them differently. A shared model then misorders some of these items for at least one client, however much data it observes. We capture this shift with a context vector, a small client state that tilts each client's labelling rule toward its preferences and is kept locally. Across two real-world datasets at opposite extremes of how items are shared between clients, we find that preferential concept shift is present in both, but that far fewer of its directions can be learned from the labels each client holds than significance testing detects. A shuffled-context control separates personalization gains from those due to added model capacity. Because the context is personal, it can also identify the client, so we ask which transmitted payload does. Moreover, we show that a contextual privacy audit of the client state links a client's sessions on both datasets, whereas the model update does so only when a client's sessions update the same shared parameters, as with tied item loadings. Keeping the state local closes the stronger channel at no cost to accuracy.