- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the author argues that existing federated LoRA methods do not adequately address cross-round adapter drift under severe data heterogeneity, which can cause majority-class collapse, and that feeding EMA-smoothed values back into both LoRA factors can hinder learning in B. The author proposes Stabilized A-Feedback (SAF), which feeds the EMA-smoothed A back into training, keeps B on the FedAvg update path, and uses EMA-averaged factors for deployment. The paper provides theoretical analysis under a simplified model and experiments across six classification tasks, reporting improved accuracy and stability over FedAvg and full EMA feedback in most evaluated settings.
Critical strengths
- The proposed method is simple to implement and adds no extra communication or client computation.
- The proposed method improves accuracy and stability in most tested settings with severe data heterogeneity.
- The authors clearly discuss the method’s limitations and report negative results.
Critical weaknesses
-
The empirical results do not clearly support the necessity of the proposed asymmetric feedback rule. The paper argues that A requires EMA feedback to prevent drift, whereas B should avoid feedback to prevent displacement squeeze. However, Table 2 shows that EMA-B, which violates both stated conditions, achieves nearly the same mean accuracy as SAF on FedIT (77.8% vs. 78.7%) and FedEx-LoRA (78.1% vs. 79.0%), and a higher mean accuracy on HetLoRA (73.6% vs. 72.1%). EMA-EVAL also achieves a slightly higher mean accuracy on LoRA-A² (76.5% vs. 75.8%). These differences should not be interpreted as statistically significant without further analysis, but they raise an important question: why is A-only feedback necessary if B-only feedback can achieve comparable performance? The authors acknowledge these results in Section 6.2 and Appendix C.11, but the explanation remains incomplete. Additional controlled experiments and direct drift measurements comparing SAF with EMA-B would help establish whether the proposed mechanism explains the observed gains, and under which conditions it applies.
-
The theoretical analysis seems too simplified to fully explain why SAF works in practice. The quantitative bounds analyze A and B separately under rank-1, single-step SGD, without explicitly tracking their coupled updates. This matters because A’s gradient depends on B, and initializing B to zero makes A’s initial task-loss gradient zero. The theory also provides no guarantee for the deployed EMA model, even though averaging B and the classification head improves accuracy by a median of 12.2 percentage points over SAF’s training-path model. The authors acknowledge these limitations, but a substantial gap remains between the theory and the reported gains.
Questions for authors
- Why does EMA-B achieve nearly the same accuracy as SAF on FedIT/FedEx-LoRA and a higher mean accuracy on HetLoRA, despite violating both proposed feedback conditions?
- Does SAF still outperform full A/B feedback when neither method feeds back the classification head? Could the authors provide this controlled comparison across the 24 settings in Table 1?
- What explains SAF’s collapse on seed 43 and its lower average accuracy than EMA on RoBERTa-large? Can the authors verify this comparison under the same software environment and clarify whether the findings extend to decoder-only LLMs and generation tasks?
- Please also refer to the weakness part.