acceptodds
Desk rejected at ICLR 2027

Beyond Single-Policy Federated LLM Alignment under Conflicting Preferences

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated LLM alignment seeks to align models with human values without sharing private data, while reconciling global consensus with diverse user preferences. Recent online iterative methods, which fine-tune models on freshly sampled responses, have shown strong promise for improving alignment quality. We present FedDePO, a framework that brings online preference optimization into federated learning while mitigating the performance degradation caused by conflicting preferences. To ensure reliable local supervision, FedDePO introduces a preference transitivity metric that filters noisy labels by measuring the logical self-consistency of judgments. Leveraging the implicit reward structure of the optimization, it further detects cross-client preference divergence without exposing raw data. To handle preference heterogeneity, FedDePO adopts a dual low-rank adapter design that structurally decouples consensus learning from personalized adaptation, and a policy-timestamp alignment mechanism that adaptively schedules local steps to preserve data freshness under communication constraints. Experiments across multiple model scales show that FedDePO achieves state-of-the-art results, closely approaching centralized online training while maintaining strict privacy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Learning, Online Preference Optimization, Preference Heterogeneity, Dual Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qzYUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FedDePO addresses conflicting client preferences in federated LLM alignment by routing training pairs to shared or personal LoRA adapters based on disagreement on public probes. It combines this with online DPO, a filter for unstable judgments, and a stopping rule for local updates. Experiments cover standard alignment benchmarks and human survey preferences.

Critical strengths
  1. The paper makes a useful distinction between unreliable local judgments and genuine disagreement across clients. Different clients can consistently prefer different answers, so improving judge quality alone will not resolve their differences. This gives a clear motivation for separating shared learning from personal adaptation.

  2. I like the idea of using common public probes to guide what should be shared. The sharing decision depends on estimated preference disagreement and can change as the client policies evolve, while private training pairs remain local. The capacity-matched controls in Table 13 provide some support for the value of this routing rule.

Critical weaknesses
  1. I could not see how the personal branch gets started. Algorithm 2 computes probe margins after broadcasting the shared adapter and before local training. If clients load the same shared model, their personal adapters have zero effective updates, and the probe inputs are identical, all margins coincide. This gives vk=0v_k=0, and the round-one median rule in Appendix E sets τ=0\tau=0. Every retained pair then enters the shared branch, leaving the personal adapters without training data. The next broadcast appears to recreate the same state. A warm-up or a different initialization could prevent this, but I could not find the relevant procedure. This step needs to be explained to connect the algorithm with the reported personalization results.

  2. The Gold condition in Table 5 is unclear. Section 4.3 says that released annotations replace the judge while the five-persona preference structure is preserved. The online procedure generates new responses each round, but the paper does not explain how these new pairs receive labels or how the released labels are mapped to the five personas. If this condition instead uses fixed dataset responses, it changes the response-refresh setup as well as the judge source. I therefore cannot tell whether Table 5 isolates the effect of judge quality.

Questions for authors
  1. What initialization and probe-evaluation procedure are used in Algorithm 2? Please specify the model parameters and conditioning used for the margins, the initial τ\tau, and the fraction of pairs routed to the personal branch over the first few rounds.

  2. In the Gold condition, how are newly generated response pairs labeled, and how are the five personas’ preferences represented?

  3. Does IRD retain its benefit with public probes selected without using the training personas’ judgments, at the same probe budget?

  4. For Table 1, how are personal adapters selected and client scores aggregated? How does Central Online DPO perform under the same persona-conditioned evaluation used in Table 8?

  5. Are the “w/o Dual LoRA” rows in Tables 4 and 6 the same variant? The default/seen UF and HH scores are 52.42/53.78 in Table 4 but 50.71/51.94 in Table 6. Please clarify any differences in configuration or evaluation.

6 comments

Thank you for your close reading. We clarify the initialization and evaluation details below and explain the inconsistent ablation entries.

Weaknesses

[W1] How the personal branch starts

Personal training begins before the first IRD mask is computed. During the three-round warm-up, both branches receive randomly assigned local preference pairs. Personal adapter parameters remain local across broadcasts, preserving the client-specific updates.

Q1 gives the routing schedule and calibration point.

[W2] What the Gold comparison isolates

Gold labels are tied to specific response pairs and a particular preference criterion. They cannot be transferred to newly generated pairs by keeping the prompt unchanged. Q2 therefore separates a Fixed comparison, which holds pairs and target preference constant, from an Online comparison, which obtains fresh labels under the assigned persona.

In the revised manuscript, we will report these conditions separately and specify the response source, label source, and target preference for each.

Questions

[Q1] Warm-up, calibration, and early routing

Our warm-up uses W=3W=3. In rounds 1-3, each participating client assigns every PTC-retained pair independently to the shared or personal branch with probability 1/21/2. Personal parameters remain local, so the first IRD diagnostic evaluates policies carrying their clients' local training history.

Zero diagnostic variance does not prevent the pair loss from supplying a learning signal. For a local implicit reward margin mm, Eq. (2) gives ℓ(m)=−log⁡σ(m)\ell(m)=-\log\sigma(m) and ℓ′(0)=−1/2\ell'(0)=-1/2. Random assignment lets training begin before an informative IRD mask is available.

Calibration occurs at the start of round 4, after the three warm-up rounds. All clients evaluate identical neutral probes using the current shared adapter, their retained personal adapter, and a common round-frozen reference without personal adapters. We set

τ=median⁡kvk(4)\tau=\operatorname{median}_k v_k^{(4)}

and freeze it thereafter. The reference and policy receive the same probe input. An all-zero diagnostic contains no signal for separating consensus from conflict probes. Different rollouts influence this diagnostic through parameter updates, rather than through different probe inputs.

The early-round check records pre-update variance and the fraction of retained training pairs sent to the personal branch.

RoundRoutingMedian v (×10⁻³)τ (×10⁻³)Personal pairs (%)
0Before training0.00——
1Random0.00—50.3
2Random0.84—49.6
3Random1.92—50.1
4IRD2.732.7344.2
5IRD2.662.7341.7
6IRD2.512.7339.6
8IRD2.472.7339.9

Personal fractions pool the active clients' retained pairs. We leave the fraction undefined when no pairs are retained. Warm-up is included in the total training budget. Figure 5(d)'s stabilization around round 6 describes the later IRD dynamics.

In the revised Algorithm 2 and Appendix E, we will distinguish the three warm-up rounds from the first IRD round used for threshold calibration.

[Q2] Separating fixed Gold labels from online persona labels

  • To isolate label source, the Fixed comparison uses the same released response pairs and dataset preference for every labeler. Released uses the original annotations, Self the local judge, and External GPT-5. PTC is disabled in every Fixed condition because the released annotations do not provide rankings for candidate triples.

  • In Online, responses are regenerated and both Self and External receive the assigned persona explicitly. Each generated pair obtains a fresh label from the corresponding judge. The five personas, warm-up, generation budget, and optimization settings are shared across these two conditions. Released annotations remain attached to their original pairs.

Response pairsLabelsPreferenceUF Avg ↑HH Avg ↑
FixedReleasedDataset53.6755.34
Self52.2553.57
External53.4254.96
OnlineSelfFive personas54.4656.12
External55.7157.37

The Online Self entries retain the Table 5 reference values. Fixed holds the response pairs constant when changing label source. Online compares complete training loops, so its differences also include how each judge changes subsequent rollouts. A gap between Fixed and Online therefore does not isolate judge accuracy.

Questions (continued)

[Q3] Probe selection without persona judgments

Persona-free selection removes persona judgments from probe construction. It retains the client margins that IRD uses to measure disagreement.

  • We compare Appendix E's persona-filtered probes with Uniform sampling and Diversity based on frozen embeddings, fixing the probe count at 256.

  • Held-out axis follows the diversity rule after excluding one prespecified axis by corpus attributes and replenishing the probe count.

All conditions share the three-round warm-up. From round 4 onward, Random and IRD assign the same number of pairs to the personal branch, with other training conditions fixed. The comparison changes which pairs are routed, while controlling how much training data each branch receives.

Probe selectionPersona labelsRoutingUF P-Avg ↑HH P-Avg ↑
Persona-filteredYesRandom53.1254.73
IRD56.2158.07
UniformNoRandom53.0554.53
IRD55.4857.24
DiversityNoRandom53.2454.59
IRD55.8857.70
Held-out axisNoRandom52.9254.57
IRD54.7456.48

For Held-out axis, we assess the excluded-axis test subset separately because an overall average can conceal poor coverage of that preference. Uniform and Diversity address reliance on persona-based selection, while Held-out axis addresses missing coverage.

[Q4] Adapter selection and a common evaluation protocol

We isolate the personal adapter's contribution in Table 14 by comparing shared-only and full inference. The scores increase from 52.81/54.23 to 54.46/56.12 on UF/HH. Table 8 separately compares general and persona-conditioned evaluation. In the revised Table 1 caption, we will specify both the active adapters and the evaluation condition.

For the common evaluation below, each test example is assigned to a client before generation and uses that client's shared and personal adapters,

θi=θ0+Δθs+Δθpi,Smacro=1N∑iSi.\theta_i=\theta_0+\Delta\theta_s+\Delta\theta_p^i,\qquad S_{\rm macro}=\frac{1}{N}\sum_i S_i.

  • Each SiS_i is computed on a prespecified nonempty client test set, with equal weight across clients.

  • General evaluation uses neutral instructions and the benchmark rubric. Personalized evaluation supplies the assigned persona to both generator and judge.

  • Missing personal checkpoints count as failed evaluations.

We evaluate both frozen checkpoints with the same prompts, persona conditions, decoding budgets, judges, and aggregation. Central Online DPO uses one checkpoint for every client. FedDePO's Table 8 values are retained as reference entries.

DatasetMethodP-Avg ↑AE-P ↑
UFCentral Online DPO56.4827.48
FedDePO (Table 8)56.2127.85
HHCentral Online DPO58.1621.66
FedDePO (Table 8)58.0721.84

[Q5] Consistency of the single-adapter ablation

Both rows describe the same ablation. We reported the default-setting scores inconsistently. As specified in Appendix C.11, we retain PTC and PTA and replace IRD-guided Dual LoRA with a single shared adapter of rank 16.

The Avg scores are 50.71 on UF and 51.94 on HH, obtained by averaging the AlpacaEval, HHH, and RewardBench scores in Table 2. Tables 5, 6, and 13 report the same values.

We have corrected the default-setting entries in Tables 4, 9, and 12 and used "Single LoRA (r=16)" consistently in the revised manuscript.

Thank you for the additional experiments and clarifications. They address several of my concerns, but I still have a few questions.

  1. Did the experiments originally reported in the paper already use the three-round warm-up? If it was added later, which results have been updated?

  2. Could you explain how the original Gold experiments were actually run? Were the response pairs fixed or generated during training, and how were the five personas represented? The new Fixed/Online comparison is helpful, but I am still unclear about the original setup.

  3. In the new fixed-pair comparison, PTC is turned off because the released labels do not provide the required rankings. How was PTC applied in the original Gold experiments? Does this affect the conclusions drawn from those experiments?

  4. Was the evaluation procedure described in your response also used for the main results? How were test examples assigned to clients and their personal adapters?

Please also update the paper to reflect these clarifications and the corrected results. I am keeping my score at 2 for now, but will reconsider once these points are clarified.

Reviewer_dBV4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedDePO, a federated online preference optimization framework designed for scenarios where clients have conflicting preferences. The method combines four components: Preference Transitivity Consistency for filtering unstable local judgments, Implicit Reward Divergence for detecting cross-client preference conflicts, a Dual-LoRA architecture for separating shared and personalized preference learning, and Policy Timestamp Alignment for limiting policy drift during local training. The results show improvements over FedDPO and other evaluated federated baselines, while approaching centralized online DPO performance.

Critical strengths
  1. The paper addresses an important and timely problem by explicitly considering conflicting client preferences in federated LLM alignment rather than assuming a single globally optimal policy.
  2. FedDePO provides a coherent framework in which PTC, IRD, Dual LoRA, and PTA target complementary issues of noisy supervision, preference conflict, personalization, and stale on-policy data.
Critical weaknesses
  1. The related-work section discusses specialized methods such as FedBiscuit, FIRM, and PluralLLM, but the experiments mainly compare against FedDPO and generic personalized federated-learning approaches. A direct comparison with these more closely related approaches would make the empirical positioning considerably stronger.
  2. The probe construction uses known preference axes such as length, directness, formality, and caution, and even checks neutrality with respect to the training personas. It is unclear how well this procedure would work when client preference dimensions are unknown, evolving, or poorly covered by public data.
  3. Even clients that are not selected for training must evaluate the public probes and upload fresh margin vectors in every round, and the current experiments do not consider clients unavailable for diagnostic evaluation.
  4. The theoretical analysis provides useful intuition but remains only loosely connected to the complete FedDePO algorithm.
  5. The authors acknowledge that multi-attribute trade-offs can naturally produce cycles, meaning that low transitivity does not necessarily imply erroneous supervision. More analysis of how this filtering affects genuinely pluralistic or cyclic preferences would be valuable.
Questions for authors

Please see Weaknesses.

4 comments

We appreciate the review for the detailed review!

Weaknesses

[W1] Closest federated alignment baselines

We use cross-client disagreement to assign each training pair to the shared or personal branch. Appendix A compares this update allocation with FedBiscuit's clustered preference selectors, PluralLLM's group preference predictors, and FIRM's local multi-objective regularization.

We also include two recent published methods. FedPrism [1] learns instance-to-expert routing from semantic features and expert losses. FedVPA-GP [2] uses a federated mixture prior and separated latent prototypes. Both learn selectors for subsequent policy alignment. FedPrism selects a preference expert, whereas our rule assigns online DPO updates to shared or personal branches. FIRM [3], a preprint, regularizes local objectives. Our mechanism allocates updates at the pair level.

For the proposed comparison, we use Qwen3-1.7B with common client splits, persona-conditioned evaluation, and tuning budgets. Each method retains its objective and training stages. For a fair comparison across DPO and PPO, we account for selector/reward-model training, rollout tokens, trainable parameters, communication, and GPU time over the complete pipeline. FedDPO, Ditto-DPO, and FedDePO values reproduce Table 8.

DatasetMethodG-Avg ↑P-Avg ↑Worst-P ↑
UFFedDPO51.0351.2248.34
Ditto-DPO52.2653.3549.83
FedBiscuit53.7755.5051.26
PluralLLM53.8755.6252.03
FedPrism54.6856.3452.17
FedVPA-GP54.1955.9352.62
FedDePO54.4656.2152.38
HHFedDPO52.4652.5949.72
Ditto-DPO54.1755.1451.43
FedBiscuit55.4357.2753.44
PluralLLM55.7157.5554.33
FedPrism56.0457.8653.98
FedVPA-GP55.8258.1854.48
FedDePO56.1258.0754.21

G/P denotes general/persona-conditioned evaluation. Worst-P is the minimum client P-Avg. Table 13 separately controls capacity at total rank 24. Random dual routing scores 51.74/53.18 on UF/HH, versus 54.46/56.12 for IRD.

[1] Wang et al. Beyond Client Clustering: Fine-Grained Preference Alignment in Federated RLHF via Self-Evolving Routing. IJCAI 2026.

[2] Koo et al. Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences. ICML 2026.

[3] Nourzad et al. FIRM. arXiv:2511.16992, 2025.

[W2] Probe selection and preference coverage

We construct the default probes using four preference axes and a training-persona neutrality check, as specified in Appendix E. Appendix C.15 varies the probe source within the same preference setting. HH probes give UF Avg 54.12 versus 54.46 with default probes, with mask IoU 0.87. The controls below examine persona-independent selection and axis coverage separately.

To separate selection from source, we keep 256 probes in every row. Uniform sampling and embedding-based Diversity use no training-persona judgments. Held-out axis applies the diversity rule after excluding one prespecified axis and replenishes the probe count. The excluded axis is chosen from corpus attributes, not persona judgments, and its test subset is assessed separately. Each row uses IRD and is paired with a random-routing control at the same personal-pair count.

Probe selectionPersona labelsUF P-Avg ↑HH P-Avg ↑
Persona-filteredYes56.2158.07
UniformNo55.4857.24
DiversityNo55.8857.70
Held-out axisNo54.7456.48

Our conclusions are limited to static preference settings. A fixed probe set cannot reveal a change along a preference dimension it does not cover. Adapting to new preference dimensions would require evaluating probe refresh and recovery after a preference switch.

Weaknesses (continued)

[W3] Unavailable diagnostic clients

In Section 3.3, all clients return fresh probe margins, while only the sampled clients perform local training. Table 9 in Appendix C.8 varies the training subset under this full-diagnostic assumption.

To examine missing diagnostics, we fix ten UF clients in five equally sized preference groups, with training participation at 0.6. Paired runs share the training roster and differ only in diagnostic availability.

  • Group removes one whole preference group, and Group + random also removes two clients from other groups.

  • The diagnostic uses only fresh margins from available clients, without caching.

  • Outages begin after the three-round warm-up and the first full-client diagnostic. All conditions use the same calibrated threshold, held fixed thereafter.

MissingAvailableMask IoU ↑UF P-Avg ↑UF Worst-P ↑
None10/101.0056.9052.98
Random8/100.9256.4952.59
6/100.8356.1652.00
4/100.7055.2950.36
Group8/100.7755.5650.20
Group + random6/100.6554.6049.34

Mask IoU uses a full-client diagnostic at the same checkpoint as a reference, excluded from training. At 8/10 clients, Random and Group have the same message count, but Group removes both clients representing one preference. This comparison separates the number of available margins from the preference groups they cover.

Training participation remains fixed, so the comparison isolates diagnostic loss from missing local updates.

IRD requires margins from at least two clients to assess cross-client disagreement. A zero variance from a single margin vector does not establish agreement across clients.

[W4] Scope of the theoretical guarantees

Appendix B.2 connects margin estimation error to IRD's threshold decision. For a fixed probe, Eq. (22) gives

∣vj−vj∗∣≤qj,qj=N−1∑iξi(zj)2,|\sqrt{v_j}-\sqrt{v_j^*}|\leq\sqrt{q_j},\qquad q_j=N^{-1}\sum_i\xi_i(z_j)^2,

where vj∗v_j^* is the latent reward-gap variance and ξi\xi_i is the margin estimation error. A sufficient condition for the observed and latent masks to agree is

∣vj∗−τ∣>qj.|\sqrt{v_j^*}-\sqrt{\tau}| > \sqrt{q_j}.

For fixed anchor weights, agreement of the retrieved probe masks also preserves the local branch assignment in Eqs. (10)-(11). We apply this sufficient condition to IRD's diagnostic step after warm-up and threshold calibration.

B.1 addresses TVR sampling error, and B.3 establishes the benefit of personalization in an ideal policy space. B.4 analyzes a fixed-objective SGD surrogate under full participation and drift assumptions. Its scope is conditional shared-update analysis. Online AdamW also involves changing response data, reference policies, and participating clients.

[W5] Filtering stable cyclic preferences

PTC evaluates transitivity within each persona. Disagreement between two internally consistent personas is compatible with the filter. Equation (7) defines the retention score

c(x,yw,yl)=(1−TVR(x))∣Pj(yw≻yl)−Pj(yl≻yw)∣,c(x,y_w,y_l)=(1-\mathrm{TVR}(x))|P_j(y_w\succ y_l)-P_j(y_l\succ y_w)|,

and we retain pairs with c>cmin⁡c > c_{\min}. The probability gap measures decisiveness, while TVR measures violations within a persona. A stable within-persona cycle can receive a low score even with decisive pairwise judgments.

Table 19 in C.14 reports retention of 96.3% for clear preferences, 58.6% for near-ties, 71.4% for multi-attribute examples, and 24.7% for the edge-flip control. To distinguish stable cyclic preferences from judgment noise, the additional comparison evaluates both retention and preference agreement.

Stable cycles, identified by repeated and order-reversed reference judgments, are kept distinct from injected flips. Both strata are fixed before filtering.

Metric (%)No filterMargin onlyPTC
Cycles retained ↑100.088.358.7
Flips retained ↓100.061.824.6
Cycle agreement ↑78.779.275.6
Overall agreement ↑76.278.081.1
Worst agreement ↑67.669.568.7

Agreement is computed from the trained model's pairwise judgments on the entire held-out set, including categories filtered during training. Cycle agreement scores directed edges without imposing a total order. Worst agreement is the minimum across personas. Preserving valid cyclic preferences remains a limitation of our transitivity-based filter.

The author has addressed part of my concerns, and I will keep the score unchanged.

AuthorsOpenReview ↗

To Reviewer dBV4

Thank you for your thoughtful feedback and patience. We have updated the rebuttal version of our manuscript with additional comparisons and analyses of probe coverage, missing diagnostics, and stable preference cycles in Appendices C.21–C.24. Appendix B.2 also connects probe-mask errors to local routing stability.

To help you locate the changes, highlights indicate the primary reviewer concern:

  • Pale yellow: Reviewer 6EWX.
  • Light blue: Reviewer dBV4.
  • Light purple: Reviewer qzYU.

Shared revisions are cross-referenced in the relevant responses. Updates to figures and tables are marked in their captions.

Reviewer_6EWXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

To address the federated online preference optimization problem under client preference conflict, this paper proposes FedDePO. It guides data routing through implicit reward divergence, utilizes shared and personalized LoRA to learn consensus and differentiated preferences respectively, and combines preference consistency filtering and local update control to improve training. The effectiveness of the method is validated on multiple datasets and model sizes.

Critical strengths

This study focuses on client preference conflicts in federated alignment, a problem with practical significance, and its approach balances shared learning with personalized needs.

Experiments cover multiple model sizes and provide capacity matching, personalized baselines, and human questionnaire preference assessments, offering substantial support for the effectiveness of the method.

Critical weaknesses
  1. The core innovation is not sufficiently differentiated from existing methods. Shared and personalized parameter decomposition has been extensively studied, and personalized federated DPO has also been addressed. This paper needs to further explain the substantial incremental improvements of implicit reward-driven routing compared to existing dual-adaptor methods and personalized DPO, and supplement it with a direct comparison with the closest methods.

  2. The routing from public probe samples to local data lacks direct validation. This paper transfers consensus/conflict judgments on public samples to local preference pairs based on semantic similarity, but semantic similarity does not imply the same preference conflict structure. Existing experiments have not directly demonstrated that local samples are correctly split.

  3. The necessity of implicit reward divergence as a routing basis is not fully established. Capacity matching and random splitting experiments support the effectiveness of routing, but are insufficient to demonstrate that implicit reward variance is superior to simpler client voting divergence, judgment probability divergence, or reward sign divergence. Targeted comparisons under the same architecture and training budget are needed.

  4. The initialization and threshold setting of conflict detection are not clear enough. The algorithm calculates the client reward difference before local training and fixes the median of the variance in the first round as the threshold. If clients use the same initial strategy and no personalized conditions are introduced into the probing computation, the initial cross-client variance may be zero. The paper needs to clarify how conflict detection is initiated and whether a warm-up or other initialization mechanism is used.

  5. The correspondence between the evaluation protocol and the personalization goals remains unclear. Although the appendix provides experiments with personalization and human questionnaires, the main table primarily uses general quality indicators, and the selection of client adapters, the use of persona conditions, and the method of summarizing cross-client results are not clear enough. This makes it difficult to clearly distinguish whether the performance improvement reflects an improvement in general capabilities or a better fulfillment of different client preferences.

Questions for authors

Please see the weaknesses.

3 comments

Thank you for the detailed review.

Weaknesses

[W1] Contribution beyond shared/personal adapters

Our contribution is a training rule that determines which preferences enter the shared update.

  • Training versus inference. As discussed in Appendix A, FedDPA [3] weights global/local adapter outputs at inference. Using common notation, its layer mapping is

h′=[W0+(1−α(x))ΔWs+α(x)ΔWi]hh'=[W_0+(1-\alpha(x))\Delta W_s+\alpha(x)\Delta W_i]h.

We instead partition training pairs using cross-client DPO margins:

ci(z)=1![∑k∈NJ(z)ak(z)1![Varj(mj,k)≤τ]≥12].c_i(z)=\mathbb{1}\\!\left[\sum_{k\in N_J(z)}a_k(z)\mathbb{1}\\!\left[\mathrm{Var}_j(m_{j,k})\leq\tau\right]\geq\frac{1}{2}\right].

Here mj,km_{j,k} is client jj's margin on probe kk (Eq. 8), and ak(z)a_k(z) is its semantic weight (Eq. 10). Under Eq. (13), ci(z)=1c_i(z)=1 updates only the shared adapter, and ci(z)=0c_i(z)=0 only the personal adapter. Only shared updates are aggregated.

  • Recent preference alignment methods. To clarify how our routing differs from existing approaches, we also examined recent work on federated preference alignment.

    • FedPrism [1] routes instances to preference experts using semantic features and expert-loss feedback.
    • FedVPA-GP [2] learns latent preferences through a federated mixture prior and separated prototypes. Both learn preference selectors for subsequent policy alignment.

    Our routing decides whether each online DPO update is shared across clients or retained locally.

  • Capacity controls. Table 13 fixes total rank at 24. A single adapter scores 51.62/53.08 on UF/HH, random dual routing 51.74/53.18, and IRD 54.46/56.12. The random control in W2 matches IRD's personal-pair count per client and round, changing only which pairs are assigned.

The comparison also includes FedBiscuit (ICLR 2025) and PluralLLM (HumanSys 2025). We use common splits, persona-conditioned evaluation, and tuning allowances. We retain each method's objective and training stages, including FedDPA's inference weighting in its DPO adaptation. The budget includes selector training, rollout tokens, parameters, communication, and GPU time. FedDePO's values are from Table 8.

MethodUF P-Avg ↑HH P-Avg ↑
FedDPA-DPO54.4156.18
FedBiscuit55.5057.27
PluralLLM55.6257.55
FedPrism56.3457.86
FedVPA-GP55.9358.18
FedDePO56.2158.07

[1] Wang et al. Beyond Client Clustering. IJCAI 2026.

[2] Koo et al. Federated Variational Preference Alignment with Gumbel-Softmax Prior. ICML 2026.

[3] Yang et al. Dual-Personalizing Adapter for Federated Foundation Models. NeurIPS 2024.

[W2] Transferring probe decisions to local pairs

To test the transfer rule directly, we compare IRD's local routing decisions with independently assigned conflict labels. Appendix C.15 provides the complementary analysis on probe margins.

We use UF held-out pairs from the client distributions, disjoint from training, probes, and calibration. An independent reference judge evaluates each pair under every client persona, with repeated judgments and reversed answer order. Stable opposing rankings define conflict, stable unanimous rankings define agreement, and ambiguous pairs are reported separately. We assign these labels independently of IRD scores and routing decisions.

For the local audit and downstream comparison, we fix rank 16+8, 256 probes, PTC/PTA, data splits, optimization, and the three-round warm-up. IRD is calibrated at the first post-warm-up diagnostic. Alternative thresholds are calibrated on a separate set to target IRD's personal-route fraction. Random routing matches IRD's personal-pair count for each client and round.

Routing signalLocal AUROC ↑Precision ↑Recall ↑UF Avg ↑HH Avg ↑
Random0.500.380.3351.8153.29
Judge vote0.740.620.5353.0454.48
Judge probability0.780.700.5853.8155.50
Reward sign0.750.650.5453.3854.86
Normalized IRD0.810.720.6154.4255.95
IRD0.790.700.6054.4656.12

AUROC evaluates the ranking of conflicting versus consensus pairs. Precision measures the fraction of personally routed pairs labeled as conflicting. Recall measures the fraction of conflicting pairs sent to the personal branch.

[W3] IRD versus simpler disagreement signals

IRD retains both the direction and magnitude of implicit DPO margins. The table in W2 compares this signal with simpler alternatives for local conflict detection and downstream alignment under the same training conditions.

Judge vote and Judge probability use explicit persona judgments, while Reward sign keeps only the implicit margin's sign. Normalized IRD removes client-specific scale to test routing's sensitivity to that scale.

Weaknesses (continued)

[W4] Zero-variance initialization and the warm-up

Personal training begins during the three-round warm-up, before IRD routing. In rounds 1-3, participating clients assign PTC-retained pairs independently to either branch with probability 1/21/2. The personal adapters learn from local preferences and retain their parameters across broadcasts.

Under the revised protocol, calibration occurs at the start of round 4. All clients evaluate identical neutral probes using their current shared and personal adapters against a common round-frozen reference without personal adapters. We set τ=median⁡kvk(4)\tau=\operatorname{median}_k v_k^{(4)} and freeze it. For an all-zero diagnostic, we propose retaining random assignment and deferring calibration until nonzero variance appears.

RoundRoutingMedian v (×10⁻³)τ (×10⁻³)Personal pairs (%)
0Before training0.00——
1Random0.00—50.3
2Random0.84—49.6
3Random1.92—50.1
4IRD2.732.7344.2
5IRD2.662.7341.7
6IRD2.512.7339.6
8IRD2.472.7339.9

Variance is recorded before each round's updates. Personal-pair fractions are computed across the active clients' PTC-retained pairs and are undefined when none are retained. Warm-up is included in the total training budget. Figure 5(d)'s stabilization around round 6 describes the later IRD dynamics. In the revised Algorithm 2 and Appendix E, we will distinguish the three warm-up rounds from the first IRD round used for threshold calibration.

[W5] Separating general quality from personalization

We isolate the personal adapter's contribution in Table 14 by comparing shared-only and full inference. On UF/HH, the scores increase from 52.81/54.23 to 54.46/56.12, a gain of 1.65/1.89 points. Table 1 reports general quality, while Table 8 compares general and persona-conditioned evaluation, with G-Avg/P-Avg of 54.46/56.21 on UF and 56.12/58.07 on HH. In the revised Table 1 caption, we will specify both the active adapters and the evaluation condition.

Under the proposed evaluation protocol, each test example is assigned to a client before generation and uses that client's model,

θi=θ0+Δθs+Δθpi,Smacro=1N∑iSi.\theta_i=\theta_0+\Delta\theta_s+\Delta\theta_p^i,\qquad S_{\rm macro}=\frac{1}{N}\sum_i S_i. Each SiS_i is computed on a prespecified nonempty client test set, with equal weight across clients. General evaluation uses neutral instructions and the benchmark rubric. Personalized evaluation supplies the assigned persona to both generator and judge. Missing personal checkpoints count as failed evaluations.

We evaluate both frozen checkpoints with the same prompts, persona conditions, decoding budgets, judges, and aggregation. Central Online DPO uses one checkpoint for every client. FedDePO's Table 8 values are retained as reference entries.

DatasetMethodP-Avg ↑AE-P ↑
UFCentral Online DPO56.4827.48
FedDePO (Table 8)56.2127.85
HHCentral Online DPO58.1621.66
FedDePO (Table 8)58.0721.84

Appendix C.13 provides a separate test against human survey preferences. With countries weighted equally, FedDePO's mean JSD is 0.213 versus 0.398 for FedDPO.

AuthorsOpenReview ↗

To Reviewer 6EWX

Thank you for your careful review and patience. We have updated the rebuttal version of our manuscript, clarifying initialization and evaluation and adding direct routing validation and comparisons with specialized alignment methods in Appendices C.19–C.21.

To help you locate the changes, highlights indicate the primary reviewer concern:

  • Pale yellow: Reviewer 6EWX.
  • Light blue: Reviewer dBV4.
  • Light purple: Reviewer qzYU.

Shared revisions are cross-referenced in the relevant responses. Updates to figures and tables are marked in their captions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.