- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes FIRM, a federated multi-objective optimization method for LLM alignment that performs regularized MGDA locally at each client. Its main idea is to reduce disagreement between client-specific objective weights, and the paper analyzes this effect through a finite-time Pareto-stationarity bound that separates weight disagreement from gradient estimation error and local model drift. The method is evaluated through PPO-based LoRA fine-tuning on helpfulness and harmlessness objectives, where the experiments report reduced weight disagreement, favorable reward trade-offs, and lower communication costs compared with selected baselines.
Critical strengths
A key strength is the theoretical analysis of multi-objective disagreement drift. The paper explicitly separates the error induced by client-specific objective weights from gradient estimation error and conventional local-model drift, and incorporates these terms into a finite-time Pareto-stationarity analysis. This provides a more structured understanding of how decentralized objective-weight selection affects federated multi-objective optimization, rather than relying solely on empirical improvements. Although the analysis is established under a theoretical actor–critic abstraction, the explicit treatment of this error source is a meaningful strength of the paper.
Critical weaknesses
Regularized MGDA has already been studied (Chen et al., 2024), so adding regularization to the client-side MGDA subproblem offers limited methodological novelty on its own. Although the convergence analysis isolates a disagreement term, it does not clearly establish how regularization reduces this disagreement or whether that reduction explains the empirical gains. Shrinking client weights toward a common center could plausibly reduce differences between clients, but it may also weaken MGDA’s ability to avoid conflicting gradients. The paper does not sufficiently examine this trade-off, leaving the mechanism behind its improvements unclear.
Lisha Chen, Heshan Fernando, Yiming Ying, and Tianyi Chen. Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance. Journal of Machine Learning Research, 25(193):1–53, 2024.
Questions for authors
How does the proposed regularized MGDA differ from existing regularized MGDA methods? More specifically, what is the key technical insight that makes this regularization particularly suited to addressing client-side objective-weight disagreement in federated optimization?
9 comments
Relationship to Chen et al. (2024)
As acknowledged in Section 3, quadratic regularization of MGDA is an established technique, including in Chen et al. (2024), and we do not claim novelty in the regularizer itself. The difference lies in the error that the regularizer controls. Chen et al. study stochastic multi-objective learning, including regularized MGDA and the interplay among optimization, generalization, and conflict avoidance. FIRM studies what happens when several clients each solve their own MGDA subproblem: their weights differ, and this difference enters the aggregated update as a separate error term, . This cross-client discrepancy is absent in a single-learner setting: for , our definition gives . Our new single-client experiment (see “Does Reduced Disagreement Help Explain the Gains?” below) supports this distinction empirically: with one client, retains a small advantage over (approximately in helpfulness and in harmlessness), whereas with multiple clients the advantage grows to approximately and .
FIRM therefore addresses the following question: when clients select weights using locally estimated gradients at evolving local policies, how do their weight differences affect the aggregated update, and how can this effect be controlled while communicating one adapted model update per client?
The key technical insight is that the sensitivity of independently solved local MGDA subproblems contributes an additional federated update error. Our analysis separates this contribution from stochastic gradient-estimation error and conventional local-model drift (Appendix F, Eq. 10). Propositions D.1–D.2 and Corollary D.3 characterize the sensitivity of the weights to gradient perturbations, with a stability constant proportional to , and Appendix F.2 carries this sensitivity into a bound on , which appears explicitly in the finite-time Pareto-stationarity bound of Theorem F.1. This characterization of how local MGDA sensitivity propagates into global federated optimization error is not addressed in Chen et al.
For completeness, the two analyses also differ in setting:
| Chen et al. (2024) | FIRM (this work) | |
|---|---|---|
| Learning setting | Centralized stochastic learning | Federated, reinforcement learning |
| Sampling | i.i.d. stochastic gradients | Markovian, with TD critic estimation error |
| Local computation | Single learner | local policy updates, heterogeneous clients |
| Role of regularization | Coefficient stability and conflict avoidance | Controls cross-client weight disagreement |
| Communication | Not applicable | One adapted model update per client |
The Trade-off with Conflict Avoidance
We agree that disagreement control should be considered alongside responsiveness to conflicting objectives. We quantify this trade-off (following Chen et al., 2024, with a slight refinement) as follows. For a fixed gradient matrix , let
\lambda_\beta = \arg\min_{\lambda\in\Delta_M} \left\\{ \\|A\lambda\\|_2^2 +\frac{\beta}{2}\\|\lambda\\|_2^2 \right\\}, \qquad d_\beta=A\lambda_\beta.
Optimality relative to the unregularized solution gives
\leq \min_{\lambda\in\Delta_M}\\|A\lambda\\|_2^2 +\frac{\beta}{2}\left(1-\frac1M\right).$$ Furthermore, testing the first-order optimality condition at each simplex vertex yields $$\langle d_\beta,g_j\rangle \geq \\|d_\beta\\|_2^2 +\frac{\beta}{2} \left( \\|\lambda_\beta\\|_2^2-\lambda_{\beta,j} \right) \geq \\|d_\beta\\|_2^2-\frac{\beta}{8}, \qquad j\in[M].$$ The final inequality follows from $$\lambda_{\beta,j}-\\|\lambda_\beta\\|_2^2 \leq \lambda_{\beta,j}-\lambda_{\beta,j}^2 \leq \frac14.$$ Under our ascent convention, unregularized MGDA satisfies $\langle d_0,g_j\rangle\geq\\|d_0\\|_2^2$, which gives the common-ascent property for updates along $d_0$ when $d_0\neq0$. Regularization therefore preserves an approximately minimum-norm direction while allowing an additive relaxation of this common-ascent inequality. This provides a quantitative counterpart to the inverse dependence on $\beta$ in the disagreement-control bound. Writing the resulting raw composite direction as $d_\beta=A\lambda_\beta$, the corresponding inequality is $$\langle d_\beta,g_j\rangle \geq \\|d_\beta\\|_2^2-\frac{\beta}{8}.$$ Thus, for sufficiently small $\beta$, the additive allowance remains small. The full derivation is now provided in Appendix J. **Empirical regularization trade-off.** The new Figure 4 compares $\beta\in\\{0,0.01,0.1,10\\}$. FIRM ($\beta=0.01$) achieves higher final helpfulness and harmlessness than both unregularized MGDA and $\beta=10$. Increasing $\beta$ to $0.1$ produces similar, slightly higher final harmlessness but lower helpfulness. For $\beta=10$, the weight trajectories remain close to $(0.5,0.5)$, approximating fixed equal weighting. Its final helpfulness is lower than that of unregularized MGDA, while their final harmlessness rewards are similar. These observations support moderate regularization as a balance between stabilization and adaptation, rather than suggesting that stronger shrinkage necessarily improves rewards.
We thank the reviewer for their thoughtful comments and valuable feedback. We have uploaded the revised paper, with the major changes highlighted in blue. Below, we clarify the relationship to prior regularized MGDA methods and provide additional theoretical and empirical evidence for FIRM.