- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In previous works, the theories on convergence of FL methods did not cover the case where the gradients are large and local smoothness grows with the gradient norm. This paper covers such a case for FedAvg with two-level normalization. The main technical challenge was how to treat the bias induced by normalization, and the paper handled this using Assumption 2.5. The paper showed convergence for various cases, and empirically showed that server-side normalization can improve the test performance of FL methods.
Critical strengths
- The paper successfully handled convergence under the generalized assumption on smoothness. The paper proposes a new framework to analyze a new problem of normalization, which is nice.
- Extensive experimental results add motivation for this paper, on why normalization technique should be used for FL methods.
Critical weaknesses
- While the paper analyzes the convergence of two-level normalization FL, I do not see theoretical gains of normalization, especially the server-side normalization. I see that the paper empirically observes the benefits of server-side normalization, but I think the benefits should be reflected inside the theories. As the paper acknowledges, normalization can make the estimation biased and affect the convergence, which is shown as the term in Theorem 3.1. But the term shows why the bias is harmful, not why server-side normalization is beneficial. Ideally, the paper should show two convergence rates respectively on the case with and without normalization. That was the most intriguing point of the paper, but unfortunately, I could not see such things.
- While the assumption on smoothness is generalized, other assumptions such as Assumption 2.5 are strong. Considering that the paper focuses on the case where the gradient norm explodes, the norms of the gradients are not properly reflected in Assumption 2.5, etc. Especially on Assumption 2.5, the paper explains the motivation for it but I still think the assumptions should consider more on the gradient norms.
- The main experiments in the main body of the paper lack the comparison with original FedAvg, which would be what most people are interested in. I see that Figure 7 shows the comparison of results with and without two-level normalization, but the performance gap between local-only normalization and two-level normalization is not large. Strangely, the test performance seems too low for ResNet-18 and CIFAR-10. CIFAR-10 integrates a relatively easy task, and I see from another paper that FedAvg can achieve 75% test accuracy even with a small CNN [1]. I understand that the performance can be varied in FL depending on the experimental settings, therefore I think the problem of the experiments is "minor". But I still think the experiments should be done under the best settings. Validation accuracies in Table 2 are also strangely low, which makes me wonder if the experimental settings were fair.
[1] Li, Yuhang, et al. "One arrow, two hawks: Sharpness-aware minimization for federated learning via global model trajectory." Forty-second International Conference on Machine Learning. 2025.
Questions for authors
- Could you show the performance of original FedAvg for FLAIR and FEMNIST?
- What part of the theorems shows the benefits of normalization, especially the server-side normalization?
- What were the experimental settings of Figure 7 that could be the reason for low test peformance of FL methods?
3 comments
Part 2/2
Could you show the performance of original FedAvg for FLAIR and FEMNIST?
See the FLAIR/FEMNIST results above (Appendix J.1, Figure 10 and Table 5).
What part of the theorems shows the benefits of normalization, especially the server-side normalization?
See the first response above (Theorems G.7 and G.2, and Proposition G.8).
What were the experimental settings of Figure 7 that could be the reason for low test peformance of FL methods?
See the experimental protocols above (Appendices A.4 and J.2).
We believe the added comparisons and controls resolve your central concerns and strengthen the original contribution. We would be grateful if you would reconsider your rating in support of acceptance.
Best regards,
The Authors
(Part 2/2)
Thank you for the detailed revision which tries to address my concerns. However, by the quite late reply of the authors, I don't have much time to parse the long revision of the manuscript; I also think that the main results should be the results of Appendix G rather than the current version, which would need quite a substantial revision. I strongly recommend re-submission to other top-tier venues, since I still think that the paper has some values.
Below shows the reason why I think it needs a substantial revision:
(i) As far as I've read, controls the difference between local-only normalization and server-side normalization, right? If is small then local-only would be worse, but how does one lower-bound ? Most FL works assume a very small learning rate, which would result in large . An experiment would be necessary to assess how truly controls the benefits of server-side normalization.
(ii) I am not convinced by the role of as pointed out by other reviewers. Isn't normalization technique devised for the case where the is large (I am not fully convinced by it, so correct me if I am wrong)? This limitation is still not addressed in the current version of the manuscript.
(iii) Clean CIFAR-10 experiments in Appendix J shows that two-level normalization is not beneficial. This might be the correct results, since might not interrupt the convergence error. Also, I think the main experiments should be done in such clean and practical settings; elsewise, the results would be non-generalizable and impractical.
Still, I think the paper has some values including the contribution on the generalized smoothness assumptions. But I think that (i) the limitation on should be addressed and (ii) Appendix G should be the main results, both of which would need a substantial revision (which is already true in the current version). I highly recommend re-submission of this paper to other top-tier venues, which would value the theoretical contributions of this paper.
Part 1/2
Dear Reviewer jkPF,
Thank you for recognizing the value of our framework under generalized smoothness. We retain the original problem setting, Algorithm 1, and core theoretical contribution; the additions directly address your questions.
We added precisely this comparison in Theorems G.7 and G.2, keeping the normalized local updates the same. Without server normalization, the guarantee controls stationarity weighted by the aggregate length rt=∣∣Δt∣∣/(Kηloc); with server normalization, it controls unweighted stationarity. Each bound uses its own trajectory. This identifies the aggregate-length attenuation removed by the server step. To also compare actual losses, Proposition G.8 proves a strict advantage over independently optimized local-only and FedAvg steps in a stochastic quadratic example with K=B=1, with coefficients chosen before the current samples.
The added analysis distinguishes large gradient magnitudes from distortion of their aggregate direction. Specifically, Theorem G.2 uses the sharper directional residual δtdir=mina≥0∣∣gˉt−aut∣∣ without Assumption 2.5. Positively collinear gradients may have arbitrarily different norms while this residual is zero, so large norm variation is compatible with informative guarantees. Theorem I.1 develops this further under relative oracle moments and client dissimilarity: conditions (54)–(55) yield arbitrary expected stationarity accuracy for fixed K with an accuracy-independent sufficient batch.
To supply the requested baselines, we added FedAvg comparisons in Appendix J.1, Figure 10 and Table 5, using 40 paired evaluation seeds. The values below are validation accuracy and AP, respectively:
The FLAIR FedAvg runs use different aggregation weights and tuning budgets. To isolate the second normalization level, the local-only/two-level comparison matches both weighting and tuning; it gives +0.305 AP points, with 95% CI [0.094,0.519].
To clarify the earlier measurements, Figure 7 uses a stress-test protocol multiplying gradients by 20 with probability 0.1, while Table 2 reports validation values from 40-round parameter selection, before 240-round testing. We complemented this protocol with 1,000-round clean CIFAR-10 runs in Appendix J.2. FedAvg reaches 78.90% IID and 75.67% non-IID, outperforming two-level in this regime. These additional runs provide a strong clean-data reference alongside the scale-outlier experiment.
(Part 1/2)