- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how federated learning affects the emergence of generalization in grokking settings. Using five established setups, the authors systematically examine the effects of client numbers, local training epochs, participation rates, data heterogeneity, and federated optimizers on generalization dynamics. They further propose Implicit Federation Bias, suggesting that aggregation preferentially suppresses client-specific structures while preserving globally consistent ones, thereby influencing the timing and emergence of generalization.
Critical strengths
- The paper offers an interesting perspective by shifting attention from final generalization performance to how generalization emerges during federated training. In particular, it shows that federation can alter the classical grokking dynamics, with training and test performance sometimes improving together after a prolonged plateau.
- The authors conduct a systematic experimental study across multiple grokking setups and federated configurations. The functional-level analysis also provides a potentially useful perspective on how aggregation influences generalization.
Critical weaknesses
My main concern is the novelty of the proposed Implicit Federation Bias. Existing studies, such as SCAFFOLD (Karimireddy et al., 2020), have already investigated client drift and the effects of inconsistent local updates on global optimization. While this paper provides an interesting functional-level interpretation, I feel that its additional insights beyond existing federated optimization theories could be more clearly established.
Questions for authors
- Could the authors further clarify how Implicit Federation Bias differs from existing theories of client drift and local-global consistency? What additional insights does it provide?
- Could these findings offer practical guidance for designing federated optimization algorithms? For example, could functional-component retention be used to improve client updates or server aggregation?
2 comments
Thank you very much for your detailed response and thoughtful clarifications. I sincerely appreciate the authors’ efforts in addressing my concerns. Given my relatively limited expertise in this specific area, and after carefully considering the perspectives and feedback from other reviewers, I would prefer to maintain my current rating while deferring to their expertise in the final assessment.
Dear Reviewer WMeJ,
Thank you for your review and for your comments on how the work shifts attention from final generalization performance to how generalization emerges during federated training and that we conduct a systematic experimental study, providing a potentially useful perspective functional-level analysis. Below we address your specific concerns and raised questions.
Novelty of findings
We acknowledge that we perhaps did not phrase parts sufficiently well leading to confusion over what is novel and as such have made the changes in the updated version.
Whilst we agree that client drift is well studied and that SCAFFOLD corrects it to improve convergence, our contribution is complementary but addresses a different question: rather than examining how client drift affects optimization convergence, we investigate which functional structures survive aggregation, and how their accumulation changes the emergence of generalization.
Our results motivate a mechanism that we call implicit federation bias, which explains that federation itself can act as a regulariser, showing behaviors such joint training-test accuracy growth, or multiple training plateaus. The main idea is that repeated local training and aggregation creates selection preference over what the global model learns. We understand that aggregation cancelling what clients disagree on is expected, what we show is how specific components contributing to memorisation and generalisation develop during training, and how different levels and factors of federation affect them.
Drift studies tend to treat the client update as one vector and do not make this distinction. We also find that drift and disagreement do not sufficiently predict when generalisation emerges and that most of the acceleration does not come from correcting drift. (Appendix E).
We have also added an additional section linking and differentiating our work from existing grokking literature (see Section 5, paragraph “Relation to other grokking explanations”).
Usage in federated optimization algorithms
We have added a causal intervention (Section 5, Figure 10, Appendix A.10). Amplifying, or keeping only, the shared component G retained by aggregation reaches generalization 1.9–3.1× sooner in Setup A. This confirms that G points towards the generalizing solution and through it we can alter the federated optimizer by boosting the learning signal on shared generalisable (or conversely memorised) components to change the learning dynamics such as to reproduce the shown joint training-test increases, to speed up convergence and produce grokking-like behaviour whilst keeping weight norms low.
However, because G relies on the task’s symmetry, this is currently diagnostic rather than for practical use; we discuss initial work and directions on G estimation to then use in optimizers.
We would like to thank the reviewer since their feedback has helped produce some additional interesting findings.
If this has addressed your concerns, we’d be grateful if you could revisit your score. We also welcome additional questions and discussion as needed.
— The Authors