acceptodds
Withdrawn from ICLR 2027

Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Second-order optimizers can significantly accelerate large-scale training, yet their naive federated variants are often unstable or even diverge on non-IID data. We show that a key culprit is preconditioner drift: client-side second-order training induces heterogeneous curvature-defined geometries (i.e., preconditioner coordinate systems), and server-side model averaging updates computed under incompatible metrics, corrupting the global descent direction. To address this geometric mismatch, we propose FedPAC, a preconditioner alignment and correction framework for reliable federated second-order optimization. FedPAC explicitly decouples parameter aggregation from geometry synchronization by: (i) Alignment (i.e.,aggregating local preconditioners into a global reference and warm-starting clients via global preconditioner); and (ii) Correction (i.e., steering local preconditioned updates using a global preconditioned direction to suppress long-term drift). We establish a drift-dependent non-convex convergence bound for an idealized preconditioned-gradient formulation of FedPAC, quantifying the interaction between preconditioner drift and data heterogeneity. Empirically, FedPAC consistently improves stability and accuracy across vision and language tasks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mMqqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper use the "preconditioned drift" to analysis why optimizer can work not well in federated learning. It proposes a FedPAC for aligning the different client state, with client side personalisation preserved. The paper also provide some theoretical proof for the analysis of the "preconditioned drift" and the convergence of the proposed FedPAC.

Critical strengths

None.

Critical weaknesses

1, the paper's discussion scope is with second-order optimizers. However, the experiments are heavily depend on the Muon, which is not a second-order optimizer. in line 2473 (the part introducing Muon), the paper writes "A momentum matrix can be rectangular and need not be symmetric or positive definite. It is therefore an optimizer state, rather than the positive-definite operator assumed in the theoretical model". It is ridiculous how the authors can leave such a displacement irresolved. Seems like an AI defensive writing without human-level resolved.

The motivations, intuition derivations are heavily based on the second-order property, but cannot fit the Muon case. Cannot distinguish that whether the experimental gain is accidentally in the special settings or general.

2, Experimentally, in Table 1, the Resnet-18 data. Fed_Muon has a more significant performance gain than other FedPAC gain. (for the Dir-0.05, form 34.76 to 54.00). It is strange and lacking convincing analysis that how a method designed for second-order optimizer work best on a non second-order optimizer.

Furthermore, table 1 has suspicious too many data entries ending with 5 or 6. Moreover, the authors do not provide the source code, and also do not provide the raw result of each seed (only report the average for all seeds).

3, Too many unnecessary maths contents, using large amounts to describe naive results. For example, (26),(28),(29) are just Cauchy. And (27) is simply the a^2+b^2 >=2ab. Please delete AI generated contents for non-academic readers, as ICLR is a serious academic conference.

Meanwhile, some AI generated formatting (i.e in line 902-950, the step 1/2/3/4 are not deleted).

Questions for authors

1, explain why choosing Muon despite the second-order scope.

2, explain why the method designed for second-order optimizer work well with non second-optimizers.

2, show the raw data with all seeds, explain why the data distribution in table 1 is strange.

Reviewer_ZXRxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies preconditioner drift as a key cause of instability in second-order optimization under non-IID data. To address this issue, the authors propose FedPAC, which aggregates and broadcasts a global reference preconditioner to align local geometry and suppress drift accumulation. The paper also establishes non-convex convergence guarantees and demonstrates improved stability and accuracy across vision and language tasks.

Critical strengths
  1. The experiments cover vision and language tasks and the reported results show that the proposed FedPAC improves stability and accuracy.
  2. The paper identifies preconditioner drift as an important source of instability when applying second-order optimizers to federated learning under non-IID data.
  3. The experimental evaluation covers multiple second-order methods, including Sophia, Muon, and SOAP. This provides evidence of the potential applicability of FedPAC beyond a single optimizer.
Critical weaknesses
  1. The authors establish a drift-dependent non-convex convergence bound for an idealized preconditioned-gradient formulation of FedPAC. The connection between the theoretical formulation and the actual optimizers is unclear.
  2. Although the experiments demonstrate promising improvements, the gains are not consistent across all evaluated configurations. As shown in table 10, FedPAC_Sophia underperforms the FedPM and Fed-Sophia on Tiny ImageNet with ResNet-18 under both IID and Dir-0.05 settings. Further analysis is needed to better understand the conditions under which FedPAC is effective and to explain the observed performance variations across different experimental settings.
Questions for authors
  1. The authors establish a drift-dependent non-convex convergence bound for an idealized preconditioned-gradient formulation of FedPAC. However, whether it holds for the actual implementations is unclear. The authors should clarify under what conditions, the theoretical guarantees apply to the practical implementations, and explicitly identify any additional assumptions required, and discuss the limitations of the current theoretical analysis.

  2. The accuracy tables report means, without standard deviations, making it difficult to assess the variability and reproducibility of the reported performance improvements. It is recommended that each experiment be conducted with at least five independent runs, with both the mean and standard deviation reported.

  3. Ablation study show that minimizing state drift alone does not maximize predictive performance and use β = 0.5 in all experiments. However, the rationale for this choice and its robustness across different datasets, heterogeneity levels, and optimizer configurations remain unclear. A more comprehensive sensitivity analysis of β would help justify this design choice.

  4. Tables 10 and 11 report result under IID and Dirichlet partitions with varying concentration parameters. FedPAC does not consistently outperform baselines across all evaluated settings, and the factors underlying these variations remain unclear. Could the authors further investigate the conditions under which preconditioner alignment and correction are beneficial?

Reviewer_t6ruOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper establishes an averaging strategy for preconditioners for second order algorithms for federated learning and illustrates its advantages in well established numerical experiments.

Critical strengths

I think the paper is relatively easy to follow (some concerns remain, see questions). The biggest strength is the general formulation of the preconditioner correction, which is applicable to many second order optimizers.

Critical weaknesses

I think the analysis establishes relatively weak guarantees relative to the current state of the art of the federated learning community - see questions. Further the experiments are at relative small scale - judging from the fact that the authors use one RTX 4090, I assume lacking HPC resources, so I only want to remark this as a potential improvement area, but not state it as a rejection criterium.

Questions for authors

Comments:

  • Sec 1: The contributions use symbols that are not defined at this stage of the paper - hard to read
  • Certain symbols are not defined at all at first occurrence, e.g. S_r - please fix this.
  • Please discuss the computational cost on client and server side, and the additional number of communication rounds and cost of the preconditioner corrections. It would be good to mark this in color somehow in the algorithm summary figure.

Questions:

  • Theorems 5.6 and 5.7 establish essentially the same drift-dependent convergence bound for FedSOA and FedPAC, with no provable reduction in the preconditioner-drift term due to the proposed alignment and correction mechanisms. Moreover, established first-order federated methods, such as FedGLOMO (Das et al., UAI 2022), obtain stronger stochastic non-convex stationarity rates under their respective assumptions.

  • (related to above) What theoretical acceleration does FedPAC provide over existing first-order federated optimization methods? In particular, can the authors establish a quantitative reduction of preconditioner drift relative to FedSOA, an improved geometry-dependent convergence constant, or a communication-complexity advantage?

  • Theorems 5.6 and 5.7 analyze an idealized linear SPD-preconditioning model, whereas the experimental implementations of Muon, Sophia, and SOAP involve momentum, nonlinear transformations, and clipping. As acknowledged in Appendix H.4, the actual update directions and synchronized optimizer states do not necessarily satisfy the theoretical assumptions. Can the authors establish that at least one of their implemented optimizers satisfies the convergence assumptions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.