- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FedDePO addresses conflicting client preferences in federated LLM alignment by routing training pairs to shared or personal LoRA adapters based on disagreement on public probes. It combines this with online DPO, a filter for unstable judgments, and a stopping rule for local updates. Experiments cover standard alignment benchmarks and human survey preferences.
Critical strengths
-
The paper makes a useful distinction between unreliable local judgments and genuine disagreement across clients. Different clients can consistently prefer different answers, so improving judge quality alone will not resolve their differences. This gives a clear motivation for separating shared learning from personal adaptation.
-
I like the idea of using common public probes to guide what should be shared. The sharing decision depends on estimated preference disagreement and can change as the client policies evolve, while private training pairs remain local. The capacity-matched controls in Table 13 provide some support for the value of this routing rule.
Critical weaknesses
-
I could not see how the personal branch gets started. Algorithm 2 computes probe margins after broadcasting the shared adapter and before local training. If clients load the same shared model, their personal adapters have zero effective updates, and the probe inputs are identical, all margins coincide. This gives , and the round-one median rule in Appendix E sets . Every retained pair then enters the shared branch, leaving the personal adapters without training data. The next broadcast appears to recreate the same state. A warm-up or a different initialization could prevent this, but I could not find the relevant procedure. This step needs to be explained to connect the algorithm with the reported personalization results.
-
The Gold condition in Table 5 is unclear. Section 4.3 says that released annotations replace the judge while the five-persona preference structure is preserved. The online procedure generates new responses each round, but the paper does not explain how these new pairs receive labels or how the released labels are mapped to the five personas. If this condition instead uses fixed dataset responses, it changes the response-refresh setup as well as the judge source. I therefore cannot tell whether Table 5 isolates the effect of judge quality.
Questions for authors
-
What initialization and probe-evaluation procedure are used in Algorithm 2? Please specify the model parameters and conditioning used for the margins, the initial , and the fraction of pairs routed to the personal branch over the first few rounds.
-
In the Gold condition, how are newly generated response pairs labeled, and how are the five personas’ preferences represented?
-
Does IRD retain its benefit with public probes selected without using the training personas’ judgments, at the same probe budget?
-
For Table 1, how are personal adapters selected and client scores aggregated? How does Central Online DPO perform under the same persona-conditioned evaluation used in Table 8?
-
Are the “w/o Dual LoRA” rows in Tables 4 and 6 the same variant? The default/seen UF and HH scores are 52.42/53.78 in Table 4 but 50.71/51.94 in Table 6. Please clarify any differences in configuration or evaluation.
6 comments
Questions (continued)
[Q3] Probe selection without persona judgments
Persona-free selection removes persona judgments from probe construction. It retains the client margins that IRD uses to measure disagreement.
-
We compare Appendix E's persona-filtered probes with Uniform sampling and Diversity based on frozen embeddings, fixing the probe count at 256.
-
Held-out axis follows the diversity rule after excluding one prespecified axis by corpus attributes and replenishing the probe count.
All conditions share the three-round warm-up. From round 4 onward, Random and IRD assign the same number of pairs to the personal branch, with other training conditions fixed. The comparison changes which pairs are routed, while controlling how much training data each branch receives.
| Probe selection | Persona labels | Routing | UF P-Avg ↑ | HH P-Avg ↑ |
|---|---|---|---|---|
| Persona-filtered | Yes | Random | 53.12 | 54.73 |
| IRD | 56.21 | 58.07 | ||
| Uniform | No | Random | 53.05 | 54.53 |
| IRD | 55.48 | 57.24 | ||
| Diversity | No | Random | 53.24 | 54.59 |
| IRD | 55.88 | 57.70 | ||
| Held-out axis | No | Random | 52.92 | 54.57 |
| IRD | 54.74 | 56.48 |
For Held-out axis, we assess the excluded-axis test subset separately because an overall average can conceal poor coverage of that preference. Uniform and Diversity address reliance on persona-based selection, while Held-out axis addresses missing coverage.
[Q4] Adapter selection and a common evaluation protocol
We isolate the personal adapter's contribution in Table 14 by comparing shared-only and full inference. The scores increase from 52.81/54.23 to 54.46/56.12 on UF/HH. Table 8 separately compares general and persona-conditioned evaluation. In the revised Table 1 caption, we will specify both the active adapters and the evaluation condition.
For the common evaluation below, each test example is assigned to a client before generation and uses that client's shared and personal adapters,
-
Each is computed on a prespecified nonempty client test set, with equal weight across clients.
-
General evaluation uses neutral instructions and the benchmark rubric. Personalized evaluation supplies the assigned persona to both generator and judge.
-
Missing personal checkpoints count as failed evaluations.
We evaluate both frozen checkpoints with the same prompts, persona conditions, decoding budgets, judges, and aggregation. Central Online DPO uses one checkpoint for every client. FedDePO's Table 8 values are retained as reference entries.
| Dataset | Method | P-Avg ↑ | AE-P ↑ |
|---|---|---|---|
| UF | Central Online DPO | 56.48 | 27.48 |
| FedDePO (Table 8) | 56.21 | 27.85 | |
| HH | Central Online DPO | 58.16 | 21.66 |
| FedDePO (Table 8) | 58.07 | 21.84 |
[Q5] Consistency of the single-adapter ablation
Both rows describe the same ablation. We reported the default-setting scores inconsistently. As specified in Appendix C.11, we retain PTC and PTA and replace IRD-guided Dual LoRA with a single shared adapter of rank 16.
The Avg scores are 50.71 on UF and 51.94 on HH, obtained by averaging the AlpacaEval, HHH, and RewardBench scores in Table 2. Tables 5, 6, and 13 report the same values.
We have corrected the default-setting entries in Tables 4, 9, and 12 and used "Single LoRA (r=16)" consistently in the revised manuscript.
Thank you for the additional experiments and clarifications. They address several of my concerns, but I still have a few questions.
-
Did the experiments originally reported in the paper already use the three-round warm-up? If it was added later, which results have been updated?
-
Could you explain how the original Gold experiments were actually run? Were the response pairs fixed or generated during training, and how were the five personas represented? The new Fixed/Online comparison is helpful, but I am still unclear about the original setup.
-
In the new fixed-pair comparison, PTC is turned off because the released labels do not provide the required rankings. How was PTC applied in the original Gold experiments? Does this affect the conclusions drawn from those experiments?
-
Was the evaluation procedure described in your response also used for the main results? How were test examples assigned to clients and their personal adapters?
Please also update the paper to reflect these clarifications and the corrected results. I am keeping my score at 2 for now, but will reconsider once these points are clarified.
Thank you for your close reading. We clarify the initialization and evaluation details below and explain the inconsistent ablation entries.
Weaknesses
Personal training begins before the first IRD mask is computed. During the three-round warm-up, both branches receive randomly assigned local preference pairs. Personal adapter parameters remain local across broadcasts, preserving the client-specific updates.
Q1 gives the routing schedule and calibration point.
Gold labels are tied to specific response pairs and a particular preference criterion. They cannot be transferred to newly generated pairs by keeping the prompt unchanged. Q2 therefore separates a Fixed comparison, which holds pairs and target preference constant, from an Online comparison, which obtains fresh labels under the assigned persona.
In the revised manuscript, we will report these conditions separately and specify the response source, label source, and target preference for each.
Questions
Our warm-up uses W=3. In rounds 1-3, each participating client assigns every PTC-retained pair independently to the shared or personal branch with probability 1/2. Personal parameters remain local, so the first IRD diagnostic evaluates policies carrying their clients' local training history.
Zero diagnostic variance does not prevent the pair loss from supplying a learning signal. For a local implicit reward margin m, Eq. (2) gives ℓ(m)=−logσ(m) and ℓ′(0)=−1/2. Random assignment lets training begin before an informative IRD mask is available.
Calibration occurs at the start of round 4, after the three warm-up rounds. All clients evaluate identical neutral probes using the current shared adapter, their retained personal adapter, and a common round-frozen reference without personal adapters. We set
τ=mediankvk(4)
and freeze it thereafter. The reference and policy receive the same probe input. An all-zero diagnostic contains no signal for separating consensus from conflict probes. Different rollouts influence this diagnostic through parameter updates, rather than through different probe inputs.
The early-round check records pre-update variance and the fraction of retained training pairs sent to the personal branch.
Personal fractions pool the active clients' retained pairs. We leave the fraction undefined when no pairs are retained. Warm-up is included in the total training budget. Figure 5(d)'s stabilization around round 6 describes the later IRD dynamics.
In the revised Algorithm 2 and Appendix E, we will distinguish the three warm-up rounds from the first IRD round used for threshold calibration.
To isolate label source, the Fixed comparison uses the same released response pairs and dataset preference for every labeler. Released uses the original annotations, Self the local judge, and External GPT-5. PTC is disabled in every Fixed condition because the released annotations do not provide rankings for candidate triples.
In Online, responses are regenerated and both Self and External receive the assigned persona explicitly. Each generated pair obtains a fresh label from the corresponding judge. The five personas, warm-up, generation budget, and optimization settings are shared across these two conditions. Released annotations remain attached to their original pairs.
The Online Self entries retain the Table 5 reference values. Fixed holds the response pairs constant when changing label source. Online compares complete training loops, so its differences also include how each judge changes subsequent rollouts. A gap between Fixed and Online therefore does not isolate judge accuracy.