- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper use the "preconditioned drift" to analysis why optimizer can work not well in federated learning. It proposes a FedPAC for aligning the different client state, with client side personalisation preserved. The paper also provide some theoretical proof for the analysis of the "preconditioned drift" and the convergence of the proposed FedPAC.
Critical strengths
None.
Critical weaknesses
1, the paper's discussion scope is with second-order optimizers. However, the experiments are heavily depend on the Muon, which is not a second-order optimizer. in line 2473 (the part introducing Muon), the paper writes "A momentum matrix can be rectangular and need not be symmetric or positive definite. It is therefore an optimizer state, rather than the positive-definite operator assumed in the theoretical model". It is ridiculous how the authors can leave such a displacement irresolved. Seems like an AI defensive writing without human-level resolved.
The motivations, intuition derivations are heavily based on the second-order property, but cannot fit the Muon case. Cannot distinguish that whether the experimental gain is accidentally in the special settings or general.
2, Experimentally, in Table 1, the Resnet-18 data. Fed_Muon has a more significant performance gain than other FedPAC gain. (for the Dir-0.05, form 34.76 to 54.00). It is strange and lacking convincing analysis that how a method designed for second-order optimizer work best on a non second-order optimizer.
Furthermore, table 1 has suspicious too many data entries ending with 5 or 6. Moreover, the authors do not provide the source code, and also do not provide the raw result of each seed (only report the average for all seeds).
3, Too many unnecessary maths contents, using large amounts to describe naive results. For example, (26),(28),(29) are just Cauchy. And (27) is simply the a^2+b^2 >=2ab. Please delete AI generated contents for non-academic readers, as ICLR is a serious academic conference.
Meanwhile, some AI generated formatting (i.e in line 902-950, the step 1/2/3/4 are not deleted).
Questions for authors
1, explain why choosing Muon despite the second-order scope.
2, explain why the method designed for second-order optimizer work well with non second-optimizers.
2, show the raw data with all seeds, explain why the data distribution in table 1 is strange.