acceptodds
Desk rejected at ICLR 2027

Preferential Federated Learning Beyond Label Calibration

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Personalized federated learning (pFL) assumes that clients differ, but usually fixes what each client keeps private before measuring how they differ. When clients are human annotators, disagreement is often treated as noise or as a difference in strictness that a per-client offset and scale can absorb. We isolate a component that no such calibration can capture: preferential concept shift, in which clients differ in their preferences, that is, in how they weigh the properties of an item, so that they label the same items but order some of them differently. A shared model then misorders some of these items for at least one client, however much data it observes. We capture this shift with a context vector, a small client state that tilts each client's labelling rule toward its preferences and is kept locally. Across two real-world datasets at opposite extremes of how items are shared between clients, we find that preferential concept shift is present in both, but that far fewer of its directions can be learned from the labels each client holds than significance testing detects. A shuffled-context control separates personalization gains from those due to added model capacity. Because the context is personal, it can also identify the client, so we ask which transmitted payload does. Moreover, we show that a contextual privacy audit of the client state links a client's sessions on both datasets, whereas the model update does so only when a client's sessions update the same shared parameters, as with tied item loadings. Keeping the state local closes the stronger channel at no cost to accuracy.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
personalized federated learning, annotator disagreement, concept shift, client heterogeneity, privacy auditing

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_6YPQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an important potential problem in today's FL personalization: preferential concept shift, in which clients weigh features in a different way, despite the labels.

Critical strengths
  • This paper studies an important potential problem in today's FL personalization: preferential concept shift, as far as I know, a missing point in today's existing FL works.
  • The illustrations and figures are easy to read and helpful (Figure 1).
  • The structure is very clean. The author well defined research questions within the contributions, and the experiment design also well covers those topics. The author also provided comprehensvie supporting materials.
Critical weaknesses
  • I generally like the idea of this paper and recognize its novelty. My major concern is the terminology used throughout the paper. The title refers to “beyond label calibration”, but it is unclear to me what exactly “label calibration” means in the context of FL. The term “calibration” first appears in Line 116, but I could not find a clear definition. In addition, another potentially confusing term is “order”, which usually implies a sequential relationship among results, although I assume this is not the intended meaning in this paper. I suggest that the authors define these terms more precisely or consider using alternative terminology.

  • With that said, personalized federated learning (pFL), rather than vanilla FL, is typically used to address client heterogeneity, including the type of concept drift considered in this paper, i.e., changes in P(Y∣X)P(Y|X). The related-work discussion appears to omit several representative FL approaches that seem relevant to this setting. To name a few:

    • FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering
    • Federated Learning with Profile Mapping under Distribution Shifts and Drifts
    • Federated Learning with Hierarchical Clustering of Local Updates to Improve Training on Non-IID Data
    • Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints

    I believe it is necessary to discuss how the proposed method relates to these baselines.

  • Following the second comment, I do not necessarily think that an extensive additional benchmark is required, since the structural assumptions differ substantially. However, I suggest that the authors clarify why IFCA is considered a sufficient representative of cluster-based personalization methods. IFCA may be a canonical baseline, but it is not necessarily representative of the current state of clustered FL under conditional heterogeneity. Would it be possible to include another, more recent baseline?

Questions for authors

Please see Critical Weaknesses.

Reviewer_FFhgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a preferential concept shift that occurs in annotation datasets where the annotators weigh the properties of an item differently. This paper proposes using a context vector on the client side to tackle this.

Critical strengths
  1. Formulation of the client scoring function ηk(x)\eta_k(x) is well motivated.
  2. Using client states (scale, level, context), we can interpret what the client model is doing in comparison to the shared model.
Critical weaknesses
  1. The performance of the proposed framework is inferior to that of baseline approaches (FL-based) in many scenarios (Table 1). Also, the paper doesn't compare the proposed method's performance with the latest personalized methods [1, 2, 3]
  2. The experimentation section is limited to a linear shared model. The paper should include comprehensive evaluation and analysis for scenarios where a deep shared model is required, such as for Tass 18, GabHate dataset [4]

References:

  1. FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering, https://openreview.net/pdf?id=kc4dZYJlJG
  2. Interaction-Aware Gaussian Weighting for Clustered Federated Learning, https://openreview.net/pdf?id=dZAQxNFKGg
  3. Personalized Federated Learning via Feature Distribution Adaptation, https://openreview.net/pdf?id=Wl2optQcng
  4. Federated Learning for Exploiting Annotators’ Disagreements in Natural Language Processing, https://aclanthology.org/2024.tacl-1.35/
Questions for authors
  1. In Figure 3(b), what does the gray bar represent?
  2. In line 372, the paper states: "the useful dimension grows as clients hold more labels". What happens if the capacity of the shared model (g) also increases?
  3. In line 413, the paper claims: "The context is therefore behavioural: it is read from what a client labels, not from who the client is.". How does the similarity between context vectors change as the number of client labels increases?
Reviewer_H4dxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates preferential concept shift in personalized federated learning, where clients may have different preferences and assign different rankings to the same items. The authors propose decomposing client-specific labeling rules into calibration and concept components to distinguish differences in labeling strictness from differences in preferences. Based on this decomposition, a low-dimensional context vector is introduced to capture client-specific preferences while keeping the corresponding parameters local. The paper further investigates how many preference directions can be reliably learned from limited client data and evaluates potential privacy risks associated with client states and model updates. Experiments on two real-world datasets demonstrate the existence of preferential concept shift and the effectiveness of the proposed decomposition.

Critical strengths
  1. The paper investigates an interesting and practically relevant problem in personalized federated learning by distinguishing preferential concept shift from conventional calibration shift. The proposed decomposition provides an interpretable way to characterize client heterogeneity, and the ordering-based statistical analysis offers useful evidence that differences in client preferences cannot always be explained by labeling strictness alone.

  2. The paper provides relatively comprehensive analyses of the proposed approach. In addition to predictive evaluations, the authors investigate the relationship between context dimensionality and available client data, conduct shuffled-context experiments to verify the effectiveness of personalization, and analyze potential privacy risks associated with transmitting client-specific information. These analyses provide useful insights into the behavior of low-dimensional personalization.

Critical weaknesses
  1. The methodological novelty of the proposed approach is relatively limited. As acknowledged in Section 2, the proposed model belongs to the well-established family of logistic factor models and low-dimensional client-specific representations. The main contribution therefore appears to lie in the interpretation and analysis of client heterogeneity rather than a substantially new federated learning method. Moreover, Table 1 shows that the proposed method achieves almost identical performance to existing client embedding and FedRep approaches on DICES-350, while Ditto generally achieves better results on R3. Although the proposed decomposition offers improved interpretability, the experiments do not demonstrate clear predictive advantages over existing personalization mechanisms.

  2. The experimental evaluation is limited in both the selection of baselines and the diversity of experimental settings. Most compared personalized federated learning methods, including IFCA, FedRep, and Ditto, were proposed around 2020–2021, while more recent approaches are largely absent. For example, PeFLL (ICLR 2024), which also investigates personalization for unseen clients, is discussed in the related work but not evaluated. FLEAD (TACL 2024), which specifically addresses annotator disagreement in federated learning, also deserves further consideration, although its learning objective differs from that of this paper. Furthermore, the experiments are conducted on only two relatively small datasets with simplified model architectures and full client participation. The baselines are also adapted to a common model family and optimization procedure rather than evaluated in their original settings. These limitations make it difficult to assess the competitiveness and generalizability of the proposed method against recent personalized federated learning approaches.

Questions for authors
  1. Since the proposed model belongs to a familiar class of low-dimensional personalization methods, could the authors further clarify its methodological advantages over existing approaches beyond interpretability? In particular, Table 1 shows nearly identical performance to client embedding and FedRep. It would also be helpful to distinguish the proposed score-level decomposition from prior shared/private low-rank personalization approaches, such as FedDecomp (ACM MM 2024), which addresses a different problem but shares the general principle of decomposing global and client-specific knowledge.

  2. Could the authors provide comparisons with more recent personalized federated learning methods, particularly approaches such as PeFLL that support personalization for unseen clients? Additionally, could the authors clarify whether FLEAD is applicable to the considered subjective annotation tasks and explain its absence from the experimental comparisons? Evaluations under more realistic federated settings, such as partial client participation and models with learned feature representations, would further strengthen the paper.

  3. Could the authors further clarify the practical privacy implications of the proposed approach? Although the client state is kept local, Table 18 shows that model updates still exhibit significant client-linkage risks on R3. Furthermore, the privacy audit does not cover the additional scalar losses and curvature summaries transmitted during optimization. It would be useful to understand whether the privacy conclusions remain valid when considering all transmitted information and repeated communication rounds.

Reviewer_4kSJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies disagreement among clients who provide subjective labels. It distinguishes calibration differences, which change prediction level and scale while preserving item order, from preferential concept shift, where clients rank some items differently. The authors propose an ordering-based characterization and statistical tests, model each client through a local level, scale, and low-dimensional context vector, and evaluate prediction and session-linkage risks on two datasets.

Critical strengths
  1. Clear distinction between strictness and preference. Separating differences that calibration can explain from those involving conflicting item rankings provides a useful perspective on client heterogeneity.
  2. The paper studies an interesting and important problem: clients’ disagreements may reflect meaningful differences in preferences rather than noise or strictness. Understanding and preserving these differences matters when learning from subjective human judgments.
  3. Thoughtful experimental controls. The shuffled-context, zeroed-context, and permuted-label experiments strengthen the evidence that the context captures client-specific predictive information.
Critical weaknesses
  1. Limited methodological novelty. The proposed model combines shared parameters with a client-specific level, scale, and low-dimensional preference vector, closely resembling existing client-embedding and private-head approaches. The authors acknowledge similar expressiveness and performance to dimension-matched baselines. The main contribution therefore lies in characterizing and testing heterogeneity, but its advance over existing analyses of annotator disagreement and preference modeling needs clearer justification.
  2. Incomplete privacy evaluation. Clients transmit scalar losses and curvature summaries in addition to model updates, but the privacy audit excludes these data-dependent messages. Keeping personal states local prevents their direct disclosure, yet does not establish privacy for the complete protocol. Secure aggregation is assumed rather than implemented.
  3. Unclear computational and communication costs. Training involves local Newton optimization, curvature computation and exchange, and loss evaluations for server-side backtracking. The paper does not quantify their overhead or establish scalability beyond the small, fully participating client populations studied. Although baselines share the optimizer, equal round counts do not necessarily imply equal communication costs.
Questions for authors
  1. What concrete modeling or deployment decision does the proposed decomposition enable that standard client embeddings or private heads cannot provide?
  2. Can you quantify the incremental benefit of preference modeling relative to its additional cost, including runtime, transmitted bytes, and communication exchanges required by backtracking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.