- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a regularization technique for FL, which increases the output margin (functional margin) from the client-side optimization. As a result, FedMargin shows a fast convergence speed compared to the original FL optimization algorithms.
Critical strengths
- The convergence speed of FedMargin seems to be much faster than the original FL algorithms, which is a nice thing.
- An observation that regularizing the margin might also increase the model complexity, seems useful for future works though it is already a known phenomenon.
Critical weaknesses
- Though the convergence speed seems to be much faster with margin maximization, the performance gain by margin maximization is only marginal, mostly showing under 1% accuracy improvement in experiments except for SVHN. In this paper, authors insist that FedMargin increases the model complexity, so the authors may be right in the sense that there could be a tradeoff between capacity and generalization, which could be a limitation of the proposed method.
- This might seem a bit far-fetched, but I am not sure if the effect on convergence speed comes from margin maximization. The observed acceleration may result from increased effective gradient magnitudes or altered optimization dynamics, rather than reduced client drift. The experiments do not sufficiently distinguish these explanations. Comparisons with learning-rate-tuned baselines and direct measurements of client drift would help validate the proposed mechanism.
- The novelty of the proposed method seems limited, since the margin losses are directly adopted from previous works. Although the general applicability of FedMargin to different FL algorithms is useful, it is unclear whether this constitutes a sufficiently substantial methodological contribution.
- Theory on client drift seems like a simple derivation from strong convexity of the loss; simplicity itself is not a weakness, but the fact that theory only provides the analysis on the client drift not the entire convergence bound may be a weakness. Also, as the paper acknowledges, Theorem B.7 is dependent on the gradient norm, so it is not guaranteed if FedMargin can surely decrease the client drift.
- Table 2 is important since it reports the best results in this paper, but the results are from single runs for each task. I think it needs more runs.
Questions for authors
- How sensitive are the results to , particularly for values below 1? Does the method consistently improve accuracy and convergence across a wider range of regularization strengths?
- Can you do more runs (5 runs) on Table 2 and report the results?