acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Aggregation Error in Federated LoRA

Abstract

Low-Rank Adaptation (LoRA) is the dominant approach for parameter-efficient federated fine-tuning, but independently averaging the two factor matrices across clients does not recover the average of their products. This aggregation incon-sistency has motivated a wave of mitigation methods—FFA-LoRA freezes one factor, RoLoRA alternates the frozen factor, FedEx-LoRA corrects the residual at the server—all built on the unstated assumption that the error is harmful. We revisit this assumption. We prove that the aggregation error equals the weighted cross-factor covariance, is bounded by the dispersion product, and decomposes into three geometric components dominated by subspace misalignment (>84%). To test whether the error causally harms accuracy, we introduce a controlled instrument: a per-client regularizer parameterized by strength λ, available in two equivalent forms (dispersion-product and output-space), used not as a deployable method but to vary error along a single axis. Across two model scales (RoBERTa-base 125M, DeBERTa-v3-large 435M), three seeds, and four orders of magnitude in λ, aggregation error varies by more than 30× while final accuracy varies by less than 1.5% among non-freezing methods. Factor-freezing methods that eliminate the error lose 1–3% accuracy on the smaller model; our instrument reduces error by 33–96% while matching unconstrained FedAvg with statistical equivalence (TOST, ∆=0.5%, α=0.05). The practical cost of aggregation inconsistency lies not in the error itself but in the constraints imposed to eliminate it.

Primary area:
optimization
Keywords:
Federated Learning, Large Language Model, Low Rank Adaptation, Robustness, Parameter Efficient Fine Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K4YROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper re-examines whether federated LoRA aggregation error impairs performance, characterizing it as cross-factor covariance and verifying this through experiments. The authors argue that, in the encoder classification tasks studied, the magnitude of the error does not directly predict accuracy.

Critical strengths
  1. The research question is valuable, distinguishing aggregation error from the training constraints introduced to eliminate it.

  2. The covariance formulation and geometric decomposition provide a clear analytical framework, facilitating an understanding of different aggregation methods.

Critical weaknesses
  1. Key error metrics are inconsistent with their theoretical definitions. Appendix H.3 acknowledges that the metrics for the DeBERTa experiments conflated the current round's global updates, resulting in non-zero values ​​for a method that should theoretically yield zero error. The absence of recalculation following state alignment directly undermines the core experimental evidence.

  2. The experiments failed to adequately isolate error from other training variations. Factors such as regularization, local steps, and re-decomposition all influence the optimization process. While FlexLoRA achieved a gain of approximately 1.49 percentage points, there is a lack of control experiments to attribute these gains to re-decomposition rather than error elimination. Consequently, the existing evidence is insufficient to support the conclusion that the error itself is harmless.

Questions for authors
  1. Do the main results still hold when recalculating the true aggregation error using the client factor means from the same update round?

  2. What direct evidence can distinguish between the error elimination gain and the refactoring gain of FlexLoRA?

1 comment

We thank the reviewer for two precise questions. We answer both directly.

Q1. Do the main results hold under the exact, same-round estimator? For the main pipeline they already are that estimator. Proposition 2 gives a check that the inflated estimator fails and the exact one passes; the main-pipeline tables pass it where both quantities are reported (Table 10: 0.037 vs. 0.216; App. H.6: 0.096 vs. 0.492), and Table 1 uses the same pipeline and estimator. So F1-F4, the PP sweep, the λ0\lambda_0 and rank ablations, and the decomposition hold as reported. The tables that fail the check are 2, 3 and 6; the cause is bfloat16 accumulation of the server average, not the anchor we described in App. H.3, which was our error (Official Comment, item 1). We respectfully disagree that this "directly undermines the core experimental evidence": it affects two supporting axes and one error column, not the 3-seed main comparison. We are re-running those settings with exact CωC_\omega and will add the bound check as a reported column so the inconsistency cannot recur.

Q2. Direct evidence separating error elimination from re-factorization in FlexLoRA. The submitted version has none; the attribution was argued, not tested. We are running the 2×22\times2 in fp32 with 3 paired seeds:

Error keptError removed
Factor-averaged parameterizationFedAvgFedEx-LoRA
SVD re-factorized parameterizationFedAvg + SVD re-factorization of AˉBˉ\bar A\bar BFlexLoRA

We will also report what the audit found about Table 3's pipeline (Official Comment, item 2), because it bears on why FlexLoRA led there. We will post all four cells when complete, attribute the gain to whichever factor the table supports, and revise the conclusion if it is error removal.

On isolating the error. For λ\lambda the paper already states the confound (Sec. 4.3). For PP, the within-row comparison (FedAvg vs. instrument at the same PP) is compute-matched. For FlexLoRA, the 2×22\times2 above is the isolation.

Reviewer_mTh3OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author argues that existing federated LoRA methods seek to eliminate aggregation error based on the insufficiently tested assumption that it harms accuracy, potentially introducing unnecessary constraints or complexity. The paper characterizes this error as a cross-factor covariance and investigates its relationship with accuracy through regularization-based interventions and experiments varying local training steps, LoRA rank, client count, and aggregation strategy. Experiments on RoBERTa-base and DeBERTa-v3-large across six GLUE tasks suggest that reducing aggregation error does not consistently improve accuracy, challenging its use as a primary target for method design.

Critical strengths
  • The paper addresses an important question: whether aggregation error in federated LoRA actually harms model accuracy.
  • The findings challenge a common assumption behind existing methods and warrant further discussion and investigation.
  • The theoretical analysis and empirical results offer a useful starting point for understanding when reducing aggregation error is beneficial.
  • I really appreciate studies that challenge common assumptions and explore counterintuitive or controversial findings, and this paper raises a question worth discussing.
Critical weaknesses
  • Changing λ affects both aggregation error and the local optimization process, so similar accuracy across different λ values does not show that the error itself is harmless, as the authors also acknowledge.
  • Increasing local steps also increases the total training budget. The accuracy gains could come from additional training despite harmful aggregation error, so this comparison does not isolate the error’s effect.
  • FlexLoRA changes both aggregation error and the factor parameterization, so it is unclear why its accuracy improves. The authors acknowledge this confound, but still attribute the gain to reparameterization without a controlled comparison.
Questions for authors

Please refer to the weakness part

2 comments

We thank the reviewer for the assessment and for stating the three confounds precisely.

λ\lambda. Agreed, and stated in Sec. 4.3 and App. G.1. This is why the conclusion is cross-checked on axes without regularization. Two of those axes were measured with an estimator that Reviewers XesY and K4YR show is not CωC_\omega (Official Comment, item 1); we are re-measuring them exactly. The main-pipeline tables pass the Prop. 2 check and stand.

Local steps. Across rows of Table 4 the budget changes with PP, as you say. Within each row FedAvg and the instrument share the budget, and the instrument's 6-96% error reductions come with Δ\DeltaAcc within ±0.4\pm0.4pp at P≥50P \ge 50. We will state in the revision which comparisons are controlled and which are only consistent.

FlexLoRA. This is the decisive confound and we are resolving it with a 2×22\times2 in fp32, 3 paired seeds: FedAvg, FedAvg + SVD re-factorization (parameterization only), FedEx-LoRA (error removal only), FlexLoRA (both). We will post the results when complete and attribute the gain to whatever the 2×22\times2 shows, followed by a revised PDF with rescoped claims.

Thanks to the authors for addressing my questions. I do not have further questions, and I lean toward keeping my original score.

Reviewer_XesYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether the federated-LoRA aggregation inconsistency (Σω_kA_kB_k ≠ ĀB̄) actually hurts accuracy. It writes the error as a weighted cross-factor covariance (Prop. 1), bounds it by the product of factor dispersions (Prop. 2), and splits it into subspace/scale/cross terms (Eq. 5). A per-client regularizer (CP or OS form) is used as a knob on the error, and the conclusion is checked on local steps, rank, client count and one exact aggregator (FlexLoRA). The paper concludes that within this setting the error is benign and that the real cost is in the constraints used to remove it.

Critical strengths
  • Paper is easy to follow.
  • The paper is honest about its limits. The scope is stated in the abstract, falsification conditions are given, estimator artifacts are disclosed (App.H.3), nd an earlier interpretation is openly retracted (App.H.4).
Critical weaknesses
  1. The causal claim is not isolated, and the one clean test points the other way. Only one axis cleanly sets the error to zero with the objective unchanged: FlexLoRA (Table 3). It gains +1.49±0.27pp, sign-consistent over three paired seeds. The paper credits this gain to re-factorization ("not to the absence of error") but runs no ablation that separates the two. The other axes don't fill the gap:

    • The λ knob reduces error by only 38% (Table 8) and leaves it flat (0.0289 - 0.0298) across three orders of magnitude of λ on DeBERTa (Table 2).
    • The ">30×" range comes from FedProx vs FedAvg, i.e. a different method.
    • P, K and r change compute, shard size or capacity along with the error.

    Effect size is a second issue. FedEx-LoRA's published RoBERTa-base GLUE gain over FedAvg is about 0.7-1.0 points, which is below the paper's own "benign" threshold of <1.5%. So the results reframe the prior gains as small more than they refute them.

  2. In three tables the reported "Agg. Err." is not Cω. Prop. 2 requires ∥Cω∥ ≤ σω(A)σω(B), but the table values break this bound: Table 3 FedAvg 0.0319 vs 0.0077, Table 6 K=20 0.0129 vs 0.0019, and Table 2 FedAvg 0.0298 vs 0.0095. App. H.3 confirms that the start-of-round anchor folds in the global product update, and FFA-LoRA, whose true error is zero, reads 0.0169. This means the DeBERTa axis and the client-count axis (the abstract's "2.5×") mostly measure something else, so the "consilience across six axes" (Sec. 6) does not stand as stated.

Questions for authors
  • All methods share η=5e-4 and FedProx uses a single µ. Since FFA-LoRA is known to need lr/α rescaling, does "factor-freezing loses 1-3%" survive a small per-method lr grid?
4 comments

We thank the reviewer for a check we should have run ourselves. We agree on one point and disagree, with evidence, on the others.

The Prop. 2 violation (we agree). The values in Tables 2, 3 and 6 are inflated by bfloat16 accumulation of the server average, as described in the Official Comment; the main-pipeline tables pass the same check (Table 10: 0.037 vs. 0.216; App. H.6: 0.096 vs. 0.492), so F1-F4, the PP sweep and the ablations stand. By the bound, the true FedAvg error in Table 3's pipeline is at most σσ=0.0077\sigma\sigma = 0.0077; our re-measurement indicates it is of order 10−310^{-3}, and we will post the exact values.

"The one clean test points the other way" (we disagree). FlexLoRA is not a clean test of the error. It changes the parameterization every round, as Reviewers mTh3 and K4YR also note, and in Table 3's pipeline that change interacts with the bf16 effect described above. Your bound makes the magnitude argument sharp. In the main pipeline, the instrument removed 0.014 of error for no accuracy change, and FedProx removed 0.035 for −0.5-0.5pp. Removing at most 0.0077 cannot plausibly yield +1.49+1.49pp through the error. We are not asking the reviewer to accept this by argument. The 2×22\times2 in the Official Comment isolates error removal (FedEx-LoRA) from re-factorization (SVD of AˉBˉ\bar A\bar B) in fp32 with 3 paired seeds, and we will post it when the seeds are complete. If exact aggregation helps, we will say so.

Effect size (partly agree). The paper does not claim exact aggregators gain nothing; it claims the error level predicts neither the sign nor the size of accuracy differences, and that the cost of factor-freezing exceeds the gain of exact aggregation. The 0.7-1.0pp the reviewer cites for FedEx-LoRA on RoBERTa-base is consistent with that reading. We will state the claim in exactly this form in the abstract.

Informativeness (we disagree). Three results were not contested in the reviews and are, to our knowledge, new: (i) the error is ≳84{\gtrsim}84% subspace misalignment (Table 9), which explains geometrically why freezing a factor is expensive; (ii) doubling the rank cuts the error ∼15×{\sim}15\times with no error control and shrinks FFA-LoRA's gap from −3.3-3.3 to −0.9-0.9pp (Table 11); (iii) FedProx's additive penalty vanishes only at δA=δB=0\delta A = \delta B = 0, whereas single-factor drift shared by all clients is error-free by Prop. 1 (F3, App. I). The audit adds a fourth, practical one: in bf16 federated LoRA at learning rates around 10−410^{-4}, the AA factor silently stops updating, so a FedAvg baseline becomes an unintended FFA-LoRA. We will document it with measurements.

PP, KK, rr (partly agree). Across rows of Table 4, yes. Within each row FedAvg and the instrument have identical compute, data and rank; the instrument reduces error by 6%, 33% and 96% at P=50,200,500P{=}50, 200, 500 with Δ\DeltaAcc of +0.2+0.2, +0.4+0.4, −0.1-0.1. That within-PP comparison is the evidence we rely on. For rr, we agree capacity and error change together and the paper says so.

">30$\times$" (agree). It is FedAvg vs. FedProx; the instrument's own multi-seed reduction is 38%. We will attribute each figure to its comparison.

FFA-LoRA learning rate (question). The evidence we have points to a rank effect rather than a learning-rate effect: on RoBERTa-base the FFA-LoRA gap shrinks from −3.3-3.3 to −0.9-0.9pp when rr doubles from 8 to 16 at the same learning rate (Table 11). A learning-rate deficit would not move with r/dr/d in this way. A per-method learning-rate grid would settle it directly, and we agree it belongs in the paper.

Thank you for the response and for being open about the bf16 findings. However, the central claim is nott supported at the strength the abstract states so I will keep my scores.

We agree that the abstract claimed more than the evidence supported, and the revised PDF (changes in blue) now states the claim at the strength we can defend. The central sentences read as follows.

"Across methods, the error varies by more than 30×30\times while accuracy among non-freezing methods varies by less than 1%1\%. FedAvg's error grows 190×190\times with local steps while its accuracy rises, and every method that reduces, removes, or re-factorizes away the error stays within 0.50.5pp of FedAvg on average over paired seeds, whereas eliminating the error by freezing a factor costs 11--33pp. Therefore, in the evaluated setting, the practical cost of aggregation inconsistency lies not in the error itself but in the constraints imposed to eliminate it, and the error level predicts neither the sign nor the size of accuracy differences. [...] We state the conditions under which our conclusion should fail and do not claim it beyond encoder classification."

The 30×30\times figure is kept because it holds in the fp32 main table, but it is now attributed to the comparison it comes from (FedProx vs. FedAvg). The claims we could not support have been withdrawn rather than softened, namely the second model scale, the client-count trend, and the attribution of FlexLoRA's gain to re-factorization. The new 2×22\times2 (Table 3) adds the one piece of evidence the submitted version lacked, exact error removal without any other change (FedEx-LoRA, 13×13\times lower delivered error, −0.1-0.1pp).

Thank you authors for the detailed explanation. However, I still believe the current manuscript falls short of the aceptance threshold.Therefore, I am keeping my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

What the reviewers found, what our audit found, and what we are doing about it

We thank the reviewers. Two reviews converge on one issue, and in auditing it we found more than the reviews asked about. We state everything here, with its scope, before the per-review replies.

  1. The Agg. Err. columns of Tables 2, 3 and 6 are not CωC_\omega. Reviewer XesY's check against Proposition 2 is correct. On inspection, the explanation we gave in App. H.3 (a start-of-round anchor) was wrong: the error estimator uses the post-round means in all pipelines. The actual cause is that the two pipelines behind these tables accumulated the server-side factor average in bfloat16. With ∣A∣F≈16|A|F \approx 16, the rounding residual alone is of order 10−210^{-2}. At the smaller learning rate of these pipelines, that residual dominates the logged value. A CPU simulation of the aggregation step reproduces Table 3's 0.0319; the exact CωC\omega in the same simulation is of order 10−310^{-3}. A second, smaller issue is that the dispersion product was logged as a product of projection means rather than per projection. The main pipeline (Tables 1, 4, 8-11) accumulates in fp32 and is unaffected. Where it reports both quantities, it passes the Prop. 2 check: Table 10 (0.037 vs. 0.216) and App. H.6 (0.096 vs. 0.492); Table 1 uses the same pipeline and estimator.
  2. A more consequential bf16 effect in the same two pipelines. The LoRA parameters there were stored in bfloat16. Near the Kaiming initialization scale, bf16 has a spacing of about 2×10−32\times10^{-3}, larger than a learning-rate 10−410^{-4} AdamW step, so most updates to the AA factor round to no change. In those pipelines FedAvg therefore trained with AA largely frozen, which lowers its accuracy. FlexLoRA's per-round SVD re-scales AA, so it escapes this effect; that gives it an advantage unrelated to aggregation error. We have re-run the paper's exact configuration with instrumentation that measures this directly, and are running the same comparison in fp32; we will post both as they complete. The DeBERTa pipeline (Table 2) shares this setting; we withdraw its error column and treat its accuracy comparison as provisional until re-run.
  3. The LS-Corr baseline was never applied. In the same audit we found that the least-squares correction raised an exception that was silently caught, so the LS-Corr rows and curves (Table 1, Figs. 1-2) are FedAvg runs. This explains why its error equaled FedAvg's. No reviewer relied on this baseline; we remove it and every statement that depended on it from the revision, and will re-run a correct version if time permits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.