acceptodds
Withdrawn from ICLR 2027

When Is Optimizer Memory Federable? Adaptive Synchronization in Federated Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated learning methods increasingly share optimizer states across clients, yet these states are synchronized on the model's clock or on a schedule fixed by the method. We ask which optimizer memories should be federated, and how often. Decoupling optimizer-memory synchronization from model synchronization, we distinguish federability, the immediate benefit of replacing a client's memory with the aggregate, from the long-run utility of a synchronization policy. Fixed-interval experiments show that Adam's second-moment tolerates long intervals, whereas the FedMuon policy we evaluate degrades once the interval is expanded under strong heterogeneity, largely through the coupling of momentum delivery with its global-direction correction. For Adam, an exact first-step analysis and measurements on training runs show that the aggregate effect of transporting the second-moment depends on the joint variation of client memory deviations and optimizer response rather than on memory discrepancy alone, and that the realized gain follows the signed alignment between the endpoint difference and the evaluation-loss gradient. We therefore introduce a matched transport-gain probe that compares short-horizon training with local versus aggregated memory, and adapt each memory's interval from its outcome. On CIFAR-10 and CIFAR-100, the controller recovers the fixed-interval sensitivities and reduces uplink by up to , including probe traffic, while staying within about one point of every-round synchronization. Code is available at the anonymous repository.

Primary area:
optimization
Keywords:
Federated learning, optimizer memory, synchronization interval, adaptive communication, Adam, Muon, communication-efficient federated learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_47uMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a method for adaptively adjusting the frequency of optimizer memory aggregation in federated learning. Specifically, the method considers Adam's second-moment estimates and FedMuon's momentum. To assess the immediate impact of optimizer memory aggregation, the authors compare the evaluation losses of two branches: one that retains the local optimizer memory and another that replaces it with the aggregated memory. The resulting loss comparison is then used to determine the synchronization schedule. The method maintains the current synchronization interval if aggregation reduces the evaluation loss and doubles the interval otherwise, while probing is performed according to a separate schedule. Experimental results demonstrate that adaptive optimizer memory synchronization reduces uplink communication costs with minimal performance degradation compared to both synchronizing every round and never synchronizing.

Critical strengths
  1. Simplicity and effectiveness. The proposed method is intuitive and straightforward to implement. It adaptively adjusts the synchronization interval across different settings without introducing substantial overhead in most cases.

  2. Decoupling optimizer memory and parameter synchronization. The idea of decoupling optimizer memory synchronization from parameter synchronization and scheduling them independently is intuitive and important.

  3. Analysis of Adam's transport gain. The paper attempts to provide a more fine-grained understanding of Adam's transport gain by decomposing it into clients' immediate responses and the signed alignment of endpoint differences.

Critical weaknesses
  1. Heavy reliance on heuristics and limited novelty. The proposed method relies heavily on heuristic design choices rather than theoretical justification or insights derived from the analysis. In essence, the method adopts a relatively straightforward approach of comparing the outcomes of synchronizing and retaining local optimizer memory to adjust the synchronization interval. As a result, the methodological novelty appears limited.

  2. Unclear necessity of adaptive synchronization. In Figure 1, the differences in test accuracy for Adam appear relatively small across synchronization intervals, except when synchronization is completely disabled. This raises questions about whether adaptive synchronization is truly necessary compared to a fixed synchronization interval. Furthermore, the main experiments in Table 1 only compare adaptive synchronization against two extreme cases: synchronizing every round and never synchronizing. Comparisons with fixed-interval synchronization strategies would be important to establish the benefits of adaptivity.

  3. Limited experimental evaluation. The experiments cover a relatively narrow range of federated learning settings, without sufficiently exploring factors such as the total number of clients, client participation ratio, and degree of data heterogeneity. Moreover, in the case of FedMuon, the observed performance differences appear to be related more to the correction schedule than to the distinction between adaptive and fixed-interval synchronization. In the ViT-Tiny experiments, the benefits of adaptive synchronization also appear to be marginal.

  4. Discrepancy between immediate federability and long-term utility. The proposed method determines synchronization intervals based solely on immediate federability, without considering the long-term utility of optimizer memory synchronization. However, an immediate reduction in evaluation loss does not necessarily imply better performance over subsequent training rounds.

Questions for authors
  1. Comparison with fixed-interval synchronization. How does the proposed adaptive synchronization method compare with fixed-interval synchronization across different interval lengths?
  2. Necessity of adaptive synchronization. As shown in Figure 1, Adam appears to be relatively insensitive to the choice of synchronization interval, as long as synchronization is performed. Given this observation, what is the advantage of adaptive synchronization over fixed-interval synchronization?
  3. Relevance of the transport gain analysis. What insights do the analyses in Sections 4.3 and 6.2 provide, and how do these insights motivate or support the design of the proposed method?
  4. Variability across random seeds. In Figure 1(c), the results appear to exhibit substantial variability across random seeds. What accounts for this variability?
Reviewer_BsjKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when optimizer states, or “optimizer memories,” should be synchronized in federated learning. The authors decouple optimizer-memory synchronization from the model synchronization clock and distinguish federability, the immediate benefit of replacing a local optimizer memory with an aggregate, from the long-run utility of a synchronization policy. They investigate Adam’s full-coordinate second moment and FedMuon’s momentum, showing that Adam’s second moment tolerates infrequent synchronization, while the evaluated FedMuon policy is highly sensitive under strong data heterogeneity, largely due to coupling momentum delivery with a global-direction correction. The paper proposes a short-horizon transport-gain probe and a simple adaptive controller that increases synchronization intervals when transport is not immediately beneficial. Experiments on CIFAR-10/100 with ResNet-18 and ViT-Tiny show reduced communication while largely preserving accuracy.

Critical strengths
  • Clear and relevant problem. The paper identifies an important but underexplored issue in federated learning: optimizer states may have different useful synchronization timescales from model parameters.
  • Good conceptual distinction. The separation between immediate “federability” and long-run policy utility is useful and clarifies why measuring optimizer-state discrepancy alone may be insufficient.
  • Technically grounded analysis. The first-step analysis for Adam’s second moment is simple but insightful, showing that aggregate transport effects depend on covariance between memory deviations and optimizer response, not just memory drift.
  • Practical adaptive mechanism. The transport-gain probe and exponential-backoff controller are easy to understand and implement.
  • Empirical results support the main thesis. The fixed-interval experiments convincingly show different synchronization sensitivities for Adam v and FedMuon M, and the adaptive controller usually recovers these patterns.
  • Useful ablation on FedMuon. The paper carefully shows that FedMuon’s sensitivity is largely caused by the correction schedule being tied to momentum delivery, which is an important diagnostic result.
  • Communication accounting includes probe traffic. This makes the reported uplink savings more credible.
Critical weaknesses
  • Experimental scope is limited. The evaluation is restricted to CIFAR-10/100, ResNet-18, and ViT-Tiny. This is narrow for claims about federated optimizer-memory synchronization, especially given motivation from larger models.
  • Controller is quite heuristic. The proposed policy only uses the sign of the short-horizon gain, never decreases the interval, and does not explicitly trade off gain magnitude against communication cost.
  • Short-horizon federability can miss long-run utility. The authors acknowledge this, and the ViT-Tiny Muon result already shows a case where the controller loses about 1.2 accuracy points. This limits confidence in the probe as a general policy signal.
  • Communication savings may be overstated in practical systems. The method reduces uplink for optimizer memory, but it adds extra client computation through probing. Wall-clock time, energy, and downlink are not fully analyzed.
  • The FedMuon conclusions are tied to one implementation. Since the sensitivity is largely due to coupling momentum delivery and correction, the conclusions may not generalize to other FedMuon variants or to a decoupled correction schedule.
  • No theoretical guarantee for the adaptive controller. The analysis explains the measured gain, but does not provide convergence or regret guarantees for the synchronization policy.
Questions for authors
  1. How would the controller perform against the best fixed interval chosen in hindsight under the same communication budget?
  2. Can the controller be modified to decrease k_x when synchronization becomes useful again later in training?
  3. How sensitive are the results to the probe horizon H, validation-set size, and probing frequency?
  4. What happens if Adam’s first moment m is also preserved or federated rather than reset every local segment?
  5. Can the authors evaluate on larger-scale or more realistic FL tasks, such as language modeling, non-CIFAR vision datasets, or cross-device simulation?
  6. How much downlink communication is required to transport optimizer memories back to clients, and does including downlink change the conclusions?
  7. For FedMuon, would an adaptive policy still be needed if momentum delivery and global-direction correction were decoupled from the start?
  8. Can the controller use the magnitude of G_r, not only its sign, to avoid missing cumulative small benefits?
Reviewer_3ui8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the synchronization of the optimizer states, i.e., optimizer memories, in Federated Learning under data heterogeneity. In this paper, it treats optimizer aggregation as a clock separate from model synchronization, which is fixed to every round with FedAvg. It distinguishes the immediate benefit of replacing local optimizer memory with an aggregate from the long-run utility of a synchronization policy. To this end, the authors study two kinds of optimizer memories: Adam's full-coordinate second moment v and FedMuon's momentum M, and analyze how memory transport changes the aggregated update and evaluation loss. Based on these observations, the authors propose a matched-branch transport-gain probe to adapt synchronization intervals online. Experiments on CIFAR-10/100 with ResNet-18 and ViT-Tiny report substantial uplink savings, while generally maintaining accuracy close to that of every-round synchronization.

Critical strengths
  • This paper aims to address an important and practical problem in federated optimization. It takes a further step beyond recent federated optimizer synchronization methods, which distinguish immediate transport benefit from long-run policy utility.
  • The idea of decoupling the optimizer-memory clock from the model clock provides an interesting and useful framework for studying federated optimizer synchronization.
  • The design of matched-branch probe control is well-motivated by some interesting observations on the aggregation of states of two kinds of optimizers.
Critical weaknesses
  • For FedMuon, the main experiments couple momentum delivery with global-direction correction. However, the experimental results reported in Section 6.1 differ substantially from the results reported in Table A4 about the gated-correction results. The authors should explain these experimental differences and clarify how strongly the ablation supports their interpretation of the main results.
  • The choice of the baselines weakens the evaluation of this paper, as the adaptive controller is primarily compared with every-round synchronization and fully local memory, leaving its advantage over simpler communication-saving schedules unclear.
  • The underlying accuracy–communication trade-off for the adaptive controller design is unclear. The controller does not explicitly account for communication cost when selecting synchronization intervals. Appendix G shows that block-mean second moments already have much lower communication cost, with similar accuracy under mild heterogeneity.
  • It is still unclear how the immediate effects of memory transport should determine the synchronization interval. Proposition 2 shows the short-horizon loss difference between transporting and retaining memory, but still lacks a clear connection to whether its sign supports the choice of synchronization interval or can predict the long-run benefit of repeated synchronization.
  • The controller heavily relies on a class-balanced labeled server validation set containing 10% of the training data. This is a strong assumption in FL. Besides, the robustness to smaller or distribution-mismatched validation sets is not evaluated.
  • The current synchronization analysis is restricted to round-wise granularity. I am curious whether sharing memory at a step-wise granularity would improve optimization enough to justify additional communication.
Questions for authors

See the concerns in the Weakness section above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.