acceptodds
Under review as a conference paper at ICLR 2027

Promoting Large Margins for Robust and Efficient Federated Learning

Abstract

We introduce FedMargin, a novel Federated Learning (FL) framework designed to enhance the performance of state-of-the-art algorithms by incorporating a large-margin promoting term into the client's loss function. By leveraging large margin - a well-established robustness measure in traditional Machine Learning - we prove that the parameter trajectories of clients are less prone to drift, leading to faster convergence and improved classification accuracy. This effect is particularly pronounced in non-i.i.d. scenarios. \method is designed to be seamlessly integrable with minimal computational and implementation overhead. An extensive evaluation on four benchmark datasets demonstrates that FedMargin consistently improves both the performance and convergence of the considered FL baseline algorithms.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Margin

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6cC4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a regularization technique for FL, which increases the output margin (functional margin) from the client-side optimization. As a result, FedMargin shows a fast convergence speed compared to the original FL optimization algorithms.

Critical strengths
  • The convergence speed of FedMargin seems to be much faster than the original FL algorithms, which is a nice thing.
  • An observation that regularizing the margin might also increase the model complexity, seems useful for future works though it is already a known phenomenon.
Critical weaknesses
  • Though the convergence speed seems to be much faster with margin maximization, the performance gain by margin maximization is only marginal, mostly showing under 1% accuracy improvement in experiments except for SVHN. In this paper, authors insist that FedMargin increases the model complexity, so the authors may be right in the sense that there could be a tradeoff between capacity and generalization, which could be a limitation of the proposed method.
  • This might seem a bit far-fetched, but I am not sure if the effect on convergence speed comes from margin maximization. The observed acceleration may result from increased effective gradient magnitudes or altered optimization dynamics, rather than reduced client drift. The experiments do not sufficiently distinguish these explanations. Comparisons with learning-rate-tuned baselines and direct measurements of client drift would help validate the proposed mechanism.
  • The novelty of the proposed method seems limited, since the margin losses are directly adopted from previous works. Although the general applicability of FedMargin to different FL algorithms is useful, it is unclear whether this constitutes a sufficiently substantial methodological contribution.
  • Theory on client drift seems like a simple derivation from strong convexity of the loss; simplicity itself is not a weakness, but the fact that theory only provides the analysis on the client drift not the entire convergence bound may be a weakness. Also, as the paper acknowledges, Theorem B.7 is dependent on the gradient norm, so it is not guaranteed if FedMargin can surely decrease the client drift.
  • Table 2 is important since it reports the best results in this paper, but the results are from single runs for each task. I think it needs more runs.
Questions for authors
  1. How sensitive are the results to λ\lambda, particularly for values below 1? Does the method consistently improve accuracy and convergence across a wider range of regularization strengths?
  2. Can you do more runs (5 runs) on Table 2 and report the results?
Reviewer_g7m9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces FedMargin, a federated learning framework designed to achieve faster convergence and improved model performance by addressing statistical heterogeneity among clients. To this end, FedMargin integrates a margin-based loss function into existing federated learning algorithms to reduce the adverse effects of non-identically distributed local data. Furthermore, this paper presents a theoretical analysis showing that, under mild assumptions, the proposed loss function increases the curvature of the optimization landscape along the SGD update direction, and argues that this property contributes to mitigating client drift and improving optimization stability.

Critical strengths

None

Critical weaknesses
  1. Limited Methodological Novelty: The proposed FedMargin method essentially combines an existing federated learning algorithm with a margin-based loss function. As such, the methodological novelty is limited, and the contribution appears to be largely incremental.
  2. Insufficient Justification for Margin-Based Loss: The paper does not clearly explain the advantages of adopting a margin-based loss over existing approaches such as FedProx, SCAFFOLD, and MOON. In particular, the authors argue in the Introduction that local training constraints can slow down learning by restricting local updates, whereas margin maximization can lead to faster convergence. However, these claims are not sufficiently supported by theoretical analysis or empirical evidence.
  3. Lack of Logical Flow and Organization: The paper lacks a coherent and systematic structure, with several abrupt transitions between paragraphs. For example, the first paragraph of the Introduction is excessively long and contains multiple central arguments without a clear focus. In contrast, the second paragraph is unusually short, while the third paragraph abruptly introduces the concept of margin maximization without establishing a logical connection to the preceding discussion. Furthermore, the rationale for placing the Related Work section after the Introduction and Methodology sections, rather than before the Methodology, is unclear.
  4. Marginal Performance Improvements and Unclear Effectiveness of Margin Loss: Contrary to the authors' claims, the experimental results do not convincingly demonstrate that the proposed margin-based loss is responsible for the reported performance improvements. The observed gains appear to be marginal, and it is difficult to determine whether they arise from the proposed margin maximization mechanism or merely from the regularization effect of introducing an additional loss term. Consequently, the experimental evidence is insufficient to establish the effectiveness of the proposed approach beyond that of a conventional regularizer.
Questions for authors

The authors are encouraged to carefully examine the weaknesses identified above. Additional questions are as follows.

  1. What are the implications of the authors' theoretical analysis of the margin loss in terms of data heterogeneity and model convergence in federated learning? In other words, what specific issues arising from data heterogeneity are addressed by the proposed theorem?
Reviewer_cheSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

FedMargin adds squared multiclass-margin losses to local classification objectives. It is evaluated as a plug-in to several FL algorithms on four datasets, reporting mostly modest accuracy gains, larger improvements on SVHN, and fewer rounds to reach fixed accuracy thresholds.

Critical strengths

The method is simple, adds no model parameters or communication, and is easy to integrate. The main accuracy comparison uses five seeds across several baselines and both IID and label-skew settings. Gains on SVHN are appreciable.

Critical weaknesses
  1. Incorrect theoretical claims. B.5 derives the claimed curvature gain 2λνi2\lambda\nu_i from the feature covariance, ignoring class-pair geometry. A valid three-class counterexample has νi=1\nu_i=1 but minimum margin curvature on ViV_i equal to 1, not at least 2. B.6 also uses 1−2ημ+η2L2≤(1−ημ)21-2\eta\mu+\eta^2L^2\le(1-\eta\mu)^2, which fails whenever L>μL>\mu and η>0\eta>0, including under its stated step-size conditions.
  2. Unresolved loss definition. If applied to logits, (1−ρ)2(1-\rho)^2 penalizes margins above 1 rather than maximizing them. Yet Fig.1 places the FedMargin peak around 5–7; at λ=3,ρ=6\lambda=3,\rho=6, the weighted max-margin penalty is 75 per sample. If the loss instead uses softmax probabilities, as in Sun et al. (2016), the large-margin interpretation is plausible, but Appendix B's linear-logit Hessian does not apply. The implementation, plotted metric and theoretical definition are not clearly aligned.
  3. The claimed mechanism is not established. Appendix B covers FedAvg's linear classification layer and assumes all updates lie in the margin-gradient subspace, which ordinary CE updates need not satisfy. A conditional local displacement bound does not establish smaller actual client drift, faster federated convergence or improved accuracy, let alone these benefits across arbitrary algorithms. No direct drift measurements are reported.
  4. Overstated framework contribution. FedMargin is presented as a framework for "any FL algorithm", but Eqs.(1)–(3) require a ground-truth class and competing-class scores. These losses do not directly apply to general regression or unsupervised FL. Although Sec.2.1 focuses on classification, the concrete contribution is a classification-loss plug-in, not an established task-agnostic FL framework; the broad compatibility claim needs explicit task and model prerequisites.
  5. Insufficient empirical support. Evaluation uses small models and only one non-IID severity. Margin weights are tuned, while comparable baseline tuning is not reported; FedProx is below FedAvg in all eight settings. The max-margin variant lowers mean accuracy for all six CIFAR-10 IID baselines, although not all decreases are significant. Speedups rely on single-run, threshold-dependent round counts rather than measured time. No adversarial, noise or OOD evaluation supports the title's robustness claim.
Questions for authors

Please refer to weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.