- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a method for adaptively adjusting the frequency of optimizer memory aggregation in federated learning. Specifically, the method considers Adam's second-moment estimates and FedMuon's momentum. To assess the immediate impact of optimizer memory aggregation, the authors compare the evaluation losses of two branches: one that retains the local optimizer memory and another that replaces it with the aggregated memory. The resulting loss comparison is then used to determine the synchronization schedule. The method maintains the current synchronization interval if aggregation reduces the evaluation loss and doubles the interval otherwise, while probing is performed according to a separate schedule. Experimental results demonstrate that adaptive optimizer memory synchronization reduces uplink communication costs with minimal performance degradation compared to both synchronizing every round and never synchronizing.
Critical strengths
-
Simplicity and effectiveness. The proposed method is intuitive and straightforward to implement. It adaptively adjusts the synchronization interval across different settings without introducing substantial overhead in most cases.
-
Decoupling optimizer memory and parameter synchronization. The idea of decoupling optimizer memory synchronization from parameter synchronization and scheduling them independently is intuitive and important.
-
Analysis of Adam's transport gain. The paper attempts to provide a more fine-grained understanding of Adam's transport gain by decomposing it into clients' immediate responses and the signed alignment of endpoint differences.
Critical weaknesses
-
Heavy reliance on heuristics and limited novelty. The proposed method relies heavily on heuristic design choices rather than theoretical justification or insights derived from the analysis. In essence, the method adopts a relatively straightforward approach of comparing the outcomes of synchronizing and retaining local optimizer memory to adjust the synchronization interval. As a result, the methodological novelty appears limited.
-
Unclear necessity of adaptive synchronization. In Figure 1, the differences in test accuracy for Adam appear relatively small across synchronization intervals, except when synchronization is completely disabled. This raises questions about whether adaptive synchronization is truly necessary compared to a fixed synchronization interval. Furthermore, the main experiments in Table 1 only compare adaptive synchronization against two extreme cases: synchronizing every round and never synchronizing. Comparisons with fixed-interval synchronization strategies would be important to establish the benefits of adaptivity.
-
Limited experimental evaluation. The experiments cover a relatively narrow range of federated learning settings, without sufficiently exploring factors such as the total number of clients, client participation ratio, and degree of data heterogeneity. Moreover, in the case of FedMuon, the observed performance differences appear to be related more to the correction schedule than to the distinction between adaptive and fixed-interval synchronization. In the ViT-Tiny experiments, the benefits of adaptive synchronization also appear to be marginal.
-
Discrepancy between immediate federability and long-term utility. The proposed method determines synchronization intervals based solely on immediate federability, without considering the long-term utility of optimizer memory synchronization. However, an immediate reduction in evaluation loss does not necessarily imply better performance over subsequent training rounds.
Questions for authors
- Comparison with fixed-interval synchronization. How does the proposed adaptive synchronization method compare with fixed-interval synchronization across different interval lengths?
- Necessity of adaptive synchronization. As shown in Figure 1, Adam appears to be relatively insensitive to the choice of synchronization interval, as long as synchronization is performed. Given this observation, what is the advantage of adaptive synchronization over fixed-interval synchronization?
- Relevance of the transport gain analysis. What insights do the analyses in Sections 4.3 and 6.2 provide, and how do these insights motivate or support the design of the proposed method?
- Variability across random seeds. In Figure 1(c), the results appear to exhibit substantial variability across random seeds. What accounts for this variability?