- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper re-examines whether federated LoRA aggregation error impairs performance, characterizing it as cross-factor covariance and verifying this through experiments. The authors argue that, in the encoder classification tasks studied, the magnitude of the error does not directly predict accuracy.
Critical strengths
-
The research question is valuable, distinguishing aggregation error from the training constraints introduced to eliminate it.
-
The covariance formulation and geometric decomposition provide a clear analytical framework, facilitating an understanding of different aggregation methods.
Critical weaknesses
-
Key error metrics are inconsistent with their theoretical definitions. Appendix H.3 acknowledges that the metrics for the DeBERTa experiments conflated the current round's global updates, resulting in non-zero values for a method that should theoretically yield zero error. The absence of recalculation following state alignment directly undermines the core experimental evidence.
-
The experiments failed to adequately isolate error from other training variations. Factors such as regularization, local steps, and re-decomposition all influence the optimization process. While FlexLoRA achieved a gain of approximately 1.49 percentage points, there is a lack of control experiments to attribute these gains to re-decomposition rather than error elimination. Consequently, the existing evidence is insufficient to support the conclusion that the error itself is harmless.
Questions for authors
-
Do the main results still hold when recalculating the true aggregation error using the client factor means from the same update round?
-
What direct evidence can distinguish between the error elimination gain and the refactoring gain of FlexLoRA?
We thank the reviewer for two precise questions. We answer both directly.
Q1. Do the main results hold under the exact, same-round estimator? For the main pipeline they already are that estimator. Proposition 2 gives a check that the inflated estimator fails and the exact one passes; the main-pipeline tables pass it where both quantities are reported (Table 10: 0.037 vs. 0.216; App. H.6: 0.096 vs. 0.492), and Table 1 uses the same pipeline and estimator. So F1-F4, the P sweep, the λ0 and rank ablations, and the decomposition hold as reported. The tables that fail the check are 2, 3 and 6; the cause is bfloat16 accumulation of the server average, not the anchor we described in App. H.3, which was our error (Official Comment, item 1). We respectfully disagree that this "directly undermines the core experimental evidence": it affects two supporting axes and one error column, not the 3-seed main comparison. We are re-running those settings with exact Cω and will add the bound check as a reported column so the inconsistency cannot recur.
Q2. Direct evidence separating error elimination from re-factorization in FlexLoRA. The submitted version has none; the attribution was argued, not tested. We are running the 2×2 in fp32 with 3 paired seeds:
We will also report what the audit found about Table 3's pipeline (Official Comment, item 2), because it bears on why FlexLoRA led there. We will post all four cells when complete, attribute the gain to whichever factor the table supports, and revise the conclusion if it is error removal.
On isolating the error. For λ the paper already states the confound (Sec. 4.3). For P, the within-row comparison (FedAvg vs. instrument at the same P) is compute-matched. For FlexLoRA, the 2×2 above is the isolation.