acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Multi-Objective Resolution in Federated LLM Alignment

Abstract

Aligning large language models (LLMs) requires balancing potentially conflicting objectives, such as helpfulness and harmlessness. Federated learning can support alignment when preference data are distributed across clients, but multi-objective alignment introduces a distinct conflict-resolution challenge. Server-side conflict resolution may require transmitting an objective-specific gradient for every client and objective, whereas independent client-side resolution can produce inconsistent objective weights because of stochastic estimation, local model drift, and client heterogeneity. We call the resulting inconsistency *multi-objective disagreement drift*. We propose **FIRM** (**F**ederated **I**n-client **R**egularized **M**ulti-objective LLM alignment), a communication-efficient client-centric framework that resolves objective conflicts locally and communicates one model update per client. FIRM regularizes each client's multiple-gradient descent (MGDA) subproblem, reducing the sensitivity of its objective weights to perturbations in estimated gradients. For a federated multi-objective actor-critic abstraction, we derive a finite-time bound on the average Pareto-stationarity gap that separates standard federated client drift from the additional disagreement induced by client-side conflict resolution. The bound establishes convergence to a residual neighborhood of Pareto stationarity and characterizes how this disagreement depends on regularization strength, batch size, local updating, and client heterogeneity. Experiments on federated LLM alignment show that FIRM reduces client objective-weight disagreement and improves the observed reward trade-off relative to the evaluated baselines. Additional experiments demonstrate stable behavior under heterogeneous reward models, stronger non-IID partitions, larger client populations, partial participation, additional objectives, and larger model scale.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Large Language Model Alignment, Federated Multi-Objective Learning, Multi-Objective LLM Alignment, LLM Post-Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pKEJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes FIRM, a federated multi-objective optimization method for LLM alignment that performs regularized MGDA locally at each client. Its main idea is to reduce disagreement between client-specific objective weights, and the paper analyzes this effect through a finite-time Pareto-stationarity bound that separates weight disagreement from gradient estimation error and local model drift. The method is evaluated through PPO-based LoRA fine-tuning on helpfulness and harmlessness objectives, where the experiments report reduced weight disagreement, favorable reward trade-offs, and lower communication costs compared with selected baselines.

Critical strengths

A key strength is the theoretical analysis of multi-objective disagreement drift. The paper explicitly separates the error induced by client-specific objective weights from gradient estimation error and conventional local-model drift, and incorporates these terms into a finite-time Pareto-stationarity analysis. This provides a more structured understanding of how decentralized objective-weight selection affects federated multi-objective optimization, rather than relying solely on empirical improvements. Although the analysis is established under a theoretical actor–critic abstraction, the explicit treatment of this error source is a meaningful strength of the paper.

Critical weaknesses

Regularized MGDA has already been studied (Chen et al., 2024), so adding regularization to the client-side MGDA subproblem offers limited methodological novelty on its own. Although the convergence analysis isolates a disagreement term, it does not clearly establish how regularization reduces this disagreement or whether that reduction explains the empirical gains. Shrinking client weights toward a common center could plausibly reduce differences between clients, but it may also weaken MGDA’s ability to avoid conflicting gradients. The paper does not sufficiently examine this trade-off, leaving the mechanism behind its improvements unclear.

Lisha Chen, Heshan Fernando, Yiming Ying, and Tianyi Chen. Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance. Journal of Machine Learning Research, 25(193):1–53, 2024.

Questions for authors

How does the proposed regularized MGDA differ from existing regularized MGDA methods? More specifically, what is the key technical insight that makes this regularization particularly suited to addressing client-side objective-weight disagreement in federated optimization?

9 comments

We thank the reviewer for their thoughtful comments and valuable feedback. We have uploaded the revised paper, with the major changes highlighted in blue. Below, we clarify the relationship to prior regularized MGDA methods and provide additional theoretical and empirical evidence for FIRM.

AuthorsOpenReview ↗

Relationship to Chen et al. (2024)

As acknowledged in Section 3, quadratic regularization of MGDA is an established technique, including in Chen et al. (2024), and we do not claim novelty in the regularizer itself. The difference lies in the error that the regularizer controls. Chen et al. study stochastic multi-objective learning, including regularized MGDA and the interplay among optimization, generalization, and conflict avoidance. FIRM studies what happens when several clients each solve their own MGDA subproblem: their weights differ, and this difference enters the aggregated update as a separate error term, Ttdis\mathcal T_t^{\mathrm{dis}}. This cross-client discrepancy is absent in a single-learner setting: for C=1C=1, our definition gives Ttdis=0\mathcal T_t^{\mathrm{dis}}=0. Our new single-client experiment (see “Does Reduced Disagreement Help Explain the Gains?” below) supports this distinction empirically: with one client, β=0.01\beta=0.01 retains a small advantage over β=0\beta=0 (approximately 0.020.02 in helpfulness and 0.010.01 in harmlessness), whereas with multiple clients the advantage grows to approximately 0.150.15 and 0.030.03.

FIRM therefore addresses the following question: when clients select weights using locally estimated gradients at evolving local policies, how do their weight differences affect the aggregated update, and how can this effect be controlled while communicating one adapted model update per client?

The key technical insight is that the sensitivity of independently solved local MGDA subproblems contributes an additional federated update error. Our analysis separates this contribution from stochastic gradient-estimation error and conventional local-model drift (Appendix F, Eq. 10). Propositions D.1–D.2 and Corollary D.3 characterize the sensitivity of the weights to gradient perturbations, with a stability constant proportional to 1/β1/\beta, and Appendix F.2 carries this sensitivity into a bound on Ttdis\mathcal T_t^{\mathrm{dis}}, which appears explicitly in the finite-time Pareto-stationarity bound of Theorem F.1. This characterization of how local MGDA sensitivity propagates into global federated optimization error is not addressed in Chen et al.

For completeness, the two analyses also differ in setting:

Chen et al. (2024)FIRM (this work)
Learning settingCentralized stochastic learningFederated, reinforcement learning
Samplingi.i.d. stochastic gradientsMarkovian, with TD critic estimation error
Local computationSingle learnerKK local policy updates, heterogeneous clients
Role of regularizationCoefficient stability and conflict avoidanceControls cross-client weight disagreement
CommunicationNot applicableOne adapted model update per client
AuthorsOpenReview ↗

The Trade-off with Conflict Avoidance

We agree that disagreement control should be considered alongside responsiveness to conflicting objectives. We quantify this trade-off (following Chen et al., 2024, with a slight refinement) as follows. For a fixed gradient matrix A=[g1,…,gM]A=[g_1,\ldots,g_M], let

\lambda_\beta = \arg\min_{\lambda\in\Delta_M} \left\\{ \\|A\lambda\\|_2^2 +\frac{\beta}{2}\\|\lambda\\|_2^2 \right\\}, \qquad d_\beta=A\lambda_\beta.

Optimality relative to the unregularized solution gives

\leq \min_{\lambda\in\Delta_M}\\|A\lambda\\|_2^2 +\frac{\beta}{2}\left(1-\frac1M\right).$$ Furthermore, testing the first-order optimality condition at each simplex vertex yields $$\langle d_\beta,g_j\rangle \geq \\|d_\beta\\|_2^2 +\frac{\beta}{2} \left( \\|\lambda_\beta\\|_2^2-\lambda_{\beta,j} \right) \geq \\|d_\beta\\|_2^2-\frac{\beta}{8}, \qquad j\in[M].$$ The final inequality follows from $$\lambda_{\beta,j}-\\|\lambda_\beta\\|_2^2 \leq \lambda_{\beta,j}-\lambda_{\beta,j}^2 \leq \frac14.$$ Under our ascent convention, unregularized MGDA satisfies $\langle d_0,g_j\rangle\geq\\|d_0\\|_2^2$, which gives the common-ascent property for updates along $d_0$ when $d_0\neq0$. Regularization therefore preserves an approximately minimum-norm direction while allowing an additive relaxation of this common-ascent inequality. This provides a quantitative counterpart to the inverse dependence on $\beta$ in the disagreement-control bound. Writing the resulting raw composite direction as $d_\beta=A\lambda_\beta$, the corresponding inequality is $$\langle d_\beta,g_j\rangle \geq \\|d_\beta\\|_2^2-\frac{\beta}{8}.$$ Thus, for sufficiently small $\beta$, the additive allowance remains small. The full derivation is now provided in Appendix J. **Empirical regularization trade-off.** The new Figure 4 compares $\beta\in\\{0,0.01,0.1,10\\}$. FIRM ($\beta=0.01$) achieves higher final helpfulness and harmlessness than both unregularized MGDA and $\beta=10$. Increasing $\beta$ to $0.1$ produces similar, slightly higher final harmlessness but lower helpfulness. For $\beta=10$, the weight trajectories remain close to $(0.5,0.5)$, approximating fixed equal weighting. Its final helpfulness is lower than that of unregularized MGDA, while their final harmlessness rewards are similar. These observations support moderate regularization as a balance between stabilization and adaptation, rather than suggesting that stronger shrinkage necessarily improves rewards.
Reviewer_p36ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies multi-objective alignment of large language models in federated settings, and proposes FIRM: each client determines its objective weights locally using regularized MGDA and uploads only one model update per round, thereby reducing communication overhead. The authors provide theoretical analysis and convergence guarantees, and validate the effectiveness of the method through experiments.

Critical strengths

The paper proposes FIRM, which resolves objective conflicts locally and communicates one model update per client, and regularizes each client's multiple-gradient descent subproblem to reduce the sensitivity of its objective weights to perturbations in the estimated gradients.

Critical weaknesses

Even if two clients obtain different objective weights, their composite gradients may still be quite similar, so using objective-weight disagreement to characterize client inconsistency may be insufficient.

Questions for authors

An adaptive adjustment scheme for β may be appropriate.

4 comments

We thank the reviewer for their thoughtful comments and valuable feedback. We have uploaded the revised paper, with the major changes highlighted in blue. Below, we address the reviewer's questions in detail.

AuthorsOpenReview ↗

W1. Objective-Weight Disagreement versus Composite-Gradient Disagreement

Figure 3 shows that regularization stabilizes client weights, and we have now extended the evaluation to examine the gradients they produce. The new results show lower average composite-gradient disagreement and fewer opposing client updates, providing evidence that the observed stabilization extends beyond the coefficients themselves.

First, at synchronized round-start probes, define:

\qquad \bar d_t=\frac1C\sum_{c=1}^{C}d_t^c, \qquad \mathcal D_{\mathrm{dir},t} =\frac1C\sum_{c=1}^{C} \\|d_t^c-\bar d_t\\|_2^2.$$ $\mathcal D_{\mathrm{dir},t}$ measures how much clients' combined gradients differ. All clients use the same global actor parameters at each probe, before the first local PPO optimizer step, while retaining client-specific data and critics. Second, we measure the fraction of rounds in which the clients' actual round-level parameter updates have negative cosine similarity. This examines opposition between the updates subsequently aggregated. **Table 2:** **Regularization and cross-client update disagreement.** Regularization reduces cross-client composite-gradient disagreement and opposing parameter updates. | $\beta$ | Negative-cosine rounds (%) | Mean probe $\mathcal D_{\mathrm{dir}}$ | | --- | --- | --- | | $0$ | $18.75 \pm 8.84$ | $21.42 \pm 8.69$ | | $0.01$ | $0.00 \pm 0.00$ | $8.42 \pm 5.63$ | On average, regularization reduces mean probe composite-gradient dispersion by approximately $61\\%$. The mean frequency of negative-cosine client updates decreases from $18.75\\%$ to zero. Thus, coefficient stabilization is accompanied by lower composite-gradient dispersion and fewer opposing round-level parameter updates. These results are now included in Table 2 of the revised paper. Together with Figure 3, these results show that the observed stabilization extends beyond coefficient agreement to lower composite-gradient dispersion and fewer opposing client updates in the evaluated runs. The reward sensitivity results in the new Figure 4 complement these measurements by examining the associated learning outcomes.
AuthorsOpenReview ↗

Q1. Adaptive Adjustment of

In FIRM, β\beta is a tunable hyperparameter that controls the trade-off between the stability of MGDA weights and their adaptation to local gradient geometry. As described in Appendix A.1, our implementation normalizes each local gradient Gram matrix by its mean diagonal before applying regularization. This accommodates changes in overall gradient scale without dynamically changing β\beta. The value of β\beta is selected empirically and remains fixed throughout each training run.

To further examine its selection, we have added a regularization-sensitivity experiment comparing β∈0,0.01,0.1,10\beta \in \\{0, 0.01, 0.1, 10\\} under matched settings. As shown in the new Figure 4 in the revised paper, β=0.01\beta=0.01 achieves the highest final helpfulness among the multi-objective configurations while maintaining harmlessness comparable to β=0.1\beta=0.1. In contrast, unregularized MGDA (β=0\beta=0) and stronger regularization (β=10\beta=10) yield less favorable reward trade-offs. These results support the fixed default in the evaluated setting. An adaptive rule that responds to changes in client-local gradient statistics could further improve this balance and is an interesting direction for future work.

Thank you for the author response. Having reviewed the author's feedback, my concerns have been addressed, and I will maintain my rating.

Reviewer_A17TOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FIRM to address the coordination of objectives, such as helpfulness and harmlessness, in federated large model alignment. The authors point out that independent multi-objective optimization on clients leads to discrepancies in objective weights. To mitigate this, they introduce a quadratic regularization term into the local MGDA to reduce the sensitivity of weights to gradient perturbations, while uploading only a single model update per round to save communication overhead. The paper provides a finite-time Pareto stationary point analysis for a theoretical variant. Experiments demonstrate that the method reduces weight discrepancies and improves the evaluated reward trade-offs. Overall, the work is well-motivated and methodologically concise. However, there remains a gap between the theory and the practical PPO implementation. Furthermore, the specific sources of the performance gains and the independent evaluation of alignment quality require further strengthening.

Critical strengths

S1. The problem identification is valuable, revealing additional sources of error distinct from traditional client drift. The paper points out that when clients independently solve for multi-objective weights, gradient sampling noise, local model variations, and task heterogeneity cause weight discrepancies, which in turn affect the aggregated updates. Distinguishing this "multi-objective discrepancy drift" from traditional parameter drift helps explain why centralized multi-objective optimization cannot be directly transferred to federated scenarios.

S2. The proposed method FIRM is concise and offers clear communication advantages. FIRM coordinates objectives internally on the client side via regularized MGDA and uploads only a single model update, making it compatible with PPO, LoRA, and standard FedAvg. Compared to server-side schemes that require uploading complete gradients for each objective, its uplink communication per round is reduced. Meanwhile, it allows clients to continuously adjust objective weights during local training, demonstrating significant practical value.

S3. The theoretical analysis aligns closely with the core mechanism, providing an interpretable error decomposition.

S4. The experiments strike a balance among overall performance, mechanism validation, and various federated configurations.

Critical weaknesses

W1. There is a gap between the theoretical analysis and the practical algorithm. The convergence analysis targets TFIRM and relies on a linear critic, full client participation, uniform aggregation, and additional temporal weight smoothing. In contrast, the actual experiments employ PPO combined with LoRA, weighted aggregation, and partial participation. Therefore, while the theory provides an explanation for the core mechanism, it does not yet directly guarantee the convergence behavior of the practical FIRM.

W2. The communication and scalability experiments have not yet fully isolated the effects of budget and configuration factors. If the authors could supplement the evaluation with comparisons under matched accuracy, identical total computational budgets, and the same participation settings, and report the cumulative communication cost required to reach the target reward, the conclusions regarding efficiency would be much more convincing.

Questions for authors

See weaknesses.

4 comments

We thank the reviewer for their thoughtful comments and valuable feedback. We have uploaded the revised paper, with the major changes highlighted in blue. Below, we address the reviewer's questions in detail. We first present the additional budget-controlled communication results addressing W2, then clarify the scope of the theoretical guarantees and their connection to practical FIRM in W1.

AuthorsOpenReview ↗

W2. Budget-Controlled Communication and Scalability Comparisons

W2. We thank the reviewer for this suggestion. To address this, we have added Table 1 in the revised manuscript, comparing FIRM, FSMGDA, and FedCMOO at matched cumulative communication budgets, including both uplink and downlink communication.

The comparison builds on the experimental controls in Section 5 and Appendix A.1. All methods use the same backbone, initialization, dataset, client partitions, generation settings, trainable parameterization, and participation settings. The complete runs match the dominant rollout and PPO training workloads, while method-specific operations introduce only a small overhead, providing approximately matched overall computational budgets. The existing reward trajectories compare methods under matched training-round and rollout budgets; the new table compares checkpoints under common cumulative communication budgets.

For this comparison, all vector payloads are accounted at FP32 precision. A single adapter-sized vector occupies 18.022418.0224 MB. Per client per round, FIRM uploads one such vector and downloads one global adapter, totaling 36.044836.0448 MB. FSMGDA uploads two objective-specific vectors and downloads one global adapter, totaling 54.067254.0672 MB. FedCMOO additionally requires a gradient-information exchange before local training so that the server can compute and return a shared objective-weight vector. Under the compression configuration, each client uploads an 18.022418.0224 MB compressed Jacobian payload for weight computation and an 18.022418.0224 MB adapted model update after local training, totaling 36.044836.0448 MB uplink. The downlink contains the 18.022418.0224 MB global adapter and two FP32 objective weights, totaling 18.02240818.022408 MB. Its total communication is therefore 54.06720854.067208 MB per client per round. FIRM resolves objectives locally and requires only the adapted model upload and global-adapter download. Thus, relative to FSMGDA and FedCMOO, FIRM uses 5050\\% less uplink communication and approximately 33.333.3\\% less total communication per round at matched precision. This complements the native-precision payload comparison already reported in the paper.

Table 1: Reward at common cumulative communication budgets per client, including uplink and downlink, with vector payloads accounted for at FP32 precision. GB denotes decimal gigabytes. Budgets are rounded upward from FIRM checkpoint costs; each method uses its latest completed checkpoint within the displayed budget.

Budget (GB/client)FIRM-HelpFIRM-HarmFSMGDA-HelpFSMGDA-HarmFedCMOO-HelpFedCMOO-Harm
0.1441800.5480.1100.4910.1130.5000.096
0.2883590.6620.1350.5350.1150.4940.111
0.4325380.7240.1480.5590.1090.5960.093
0.5767170.7690.1540.5520.1230.6080.079

FIRM achieves the highest helpfulness at every reported budget and higher rewards on both objectives from 0.2883590.288359 GB per client onward. Besides its smaller communication payload, FIRM allows the local policy and objective weights to adapt together within each communication round. Clients generate subsequent rollout batches using their updated policies and recompute regularized MGDA weights from the current local gradient estimates.

FSMGDA instead constructs objective-specific local updates whose multi-objective combination is determined at the server after local training. FedCMOO updates policies locally using a shared server-computed weight vector that remains fixed within the round. FIRM's within-round coefficient updates can therefore respond to changes in local gradient geometry as the policy evolves.

For a joint target of helpfulness ≥0.60\geq0.60 and harmlessness ≥0.12\geq0.12, FIRM satisfies both thresholds at the budget of 0.2883590.288359 GB per client. Neither baseline satisfies both thresholds at any of the budgets through 0.5767170.576717 GB per client. Moreover, FIRM at 0.2883590.288359 GB achieves higher rewards on both objectives than either baseline at 0.5767170.576717 GB, using approximately 5050\\% less communication at these checkpoints. This reward-dependent comparison is distinct from the structural per-round payload saving: it reflects both communication per round and the learning progress achieved. Together, the results demonstrate a favorable reward–communication trade-off in the evaluated setting.

We also clarify the role of the additional scalability experiments. The sixteen-client experiment evaluates robustness to a larger client population (Appendix A.3.2). The partial-participation experiment maintains eight active clients per round while sampling from a larger client population, preserving the per-round rollout budget and FIRM communication volume (Appendix A.3.7). These experiments assess robustness to changes in the federated configuration, while the new budget-controlled comparison provides direct evidence of communication efficiency under matched participation settings.

AuthorsOpenReview ↗

W1. Relationship Between TFIRM's Analysis and Practical FIRM

W1 The finite-time convergence guarantee applies to TFIRM, which assumes a linear critic, full participation, uniform aggregation, and temporal weight smoothing; it is not a direct convergence guarantee for PPO+LoRA training. TFIRM provides a tractable setting for isolating multi-objective disagreement drift and characterizing how regularization controls its contribution to the aggregated update. The analysis permits nonlinear policy parameterizations under the stated smoothness and boundedness assumptions and explicitly incorporates critic approximation and estimation errors.

Importantly, Proposition D.2 establishes stability of the regularized MGDA coefficient-selection map independently of those critic, participation, aggregation, and temporal-smoothing assumptions. It bounds coefficient sensitivity to perturbations in the Gram matrix supplied to the solver. Appendix F.2 incorporates this property into the federated actor–critic analysis underlying Theorem F.1.

Our experiments provide complementary evidence in the practical PPO/LoRA setting. RQ2 / Figure 3 demonstrates reduced cross-client coefficient disagreement. The practical experiments use weighted aggregation, and Appendix A.3.7 additionally evaluates partial participation. These results examine empirical behavior beyond the theorem's setting.

Thanks for the authors' response. Most of my concerns have been addressed. I would like to keep my rating as Weak Acceptance.

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your time and feedback. As the discussion period draws to a close, please take a moment to reply to the authors if you have yet to comment on their response or follow up on their latest reply.

Please let them know which concerns have been addressed and which, if any, remain. If you have no further questions, a brief confirmation is sufficient.

Best,
Your AC

AuthorsOpenReview ↗

Summary of Revisions and Additional Results

We thank all the reviewers for their constructive feedback. We appreciate that reviewers highlighted the identification of multi-objective disagreement drift as an additional error source distinct from classical client drift (A17T, pKEJ), the finite-time analysis separating disagreement from gradient-estimation error and local-model drift (A17T, pKEJ), and FIRM's communication-efficient design that resolves objective conflicts locally while transmitting a single model update per client (A17T, p36Z).

We have uploaded a revised manuscript, with new content highlighted in blue; minor editorial corrections are not highlighted. As the discussion period approaches its end, we would like to briefly summarize the main additions:

New baseline: We implemented DREAM (Zhou et al., 2026) in our PPO/LoRA setting and added performance and communication-budget comparisons, along with a discussion of its relationship to FIRM (Appendix A.5).

Regularization mechanism: New composite-gradient dispersion and round-level update-opposition measurements (Section 5, Table 2; Appendix A.2.1), a single-client comparison (Appendix A.2.2), and a sensitivity study over β\beta (Figure 4).

Stability–adaptivity trade-off: A theoretical bound characterizing the cost of regularization relative to unregularized MGDA (Lemma J.1, Appendix J).

Communication efficiency: Reward comparisons under matched cumulative communication budgets (Tables 1 and 4), with detailed communication accounting (Appendix A.6).

Related work and presentation: Expanded discussion of relevant methods (Appendix B), improved writing, and corrected typos and editorial issues.

We hope these revisions address the reviewers’ concerns and help clarify the paper’s contributions. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.