acceptodds
Under review as a conference paper at ICLR 2027

Consistent Subspace Aggregation for Personalized Federated LoRA

Abstract

Federated fine-tuning with LoRA suffers from an aggregation inconsistency, as averaging low-rank factors does not recover the average of the updates they induce. Existing remedies need both factors at the server, but personalized protocols share only the right factors and keep client-specific left factors local, where factor averaging depends on each client's arbitrary factorization. We propose PERSIA, a fully low-rank aggregation rule for this setting, where each client whitens its right factor with the Gram matrix of its private left factor, and the server recovers a shared right subspace by truncated SVD. We prove that the core PERSIA rule exactly minimizes the clients' reconstruction error among all shared rank- right factors, with an objective invariant to equivalent factorizations, and give a factor-stationarity bound whose floor does not grow with the number of local steps. On GLUE with RoBERTa-Large, right after aggregating a 1280-step local phase, PERSIA's personalized models are as accurate as the clients' local models (91.1% versus 90.8%), whereas factor averaging loses 4.1 points and two-factor FedAvg 11.3 points; after further rounds, final accuracy is on par across rules, with PERSIA uploading half as much as two-factor sharing.

Primary area:
optimization
Keywords:
Federated Learning, Personalization, Low-rank Adaptation, Model Merging

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_GXNdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PERSIA, a consistent subspace aggregation method for personalized federated LoRA, where clients share the right factors while keeping their left factors local. PERSIA uses Gram-based transformations and truncated SVD to construct a shared right subspace, followed by client-side rebasing to preserve personalized updates. The authors establish an optimality guarantee for reconstruction error and provide a factor-stationarity analysis.

Critical strengths

Over longer local training intervals, PERSIA significantly mitigated the decline in post-aggregation instantaneous accuracy, validating the effectiveness of the proposed mechanism in improving aggregation consistency.

Critical weaknesses

1.The method’s motivation and practical necessity need to be further strengthened. This paper is based on the one-sided parameter-sharing setup of FedSA-LoRA; however, given that the communication overhead of the B matrix in LoRA is relatively limited, the practical benefits of omitting the transmission of B still require more thorough justification. Furthermore, the design of this method relies on SVD reconstruction on the server side, but the validity of this objective decomposition still needs further explanation.

2.The overall performance gains and robustness are limited. Although PERSIA demonstrates outstanding performance in terms of real-time accuracy after aggregation, its overall performance did not outperform existing baselines in multiple experiments. The method’s performance deteriorates under more heterogeneous client conditions. Experimental comparisons with representative recent federated fine-tuning methods remain limited.

Questions for authors
  1. The authors need to further explain the rationale for decomposition on the server side and why they chose to retain the V-matrix.
  2. The authors need to provide additional explanation regarding the robustness of the experimental results.
6 comments
AuthorsOpenReview ↗

Response to Reviewer GXNd (1/3)

We thank the reviewer for the careful assessment and for recognizing that PERSIA significantly mitigated the decline in post-aggregation instantaneous accuracy, validating the effectiveness of the proposed mechanism in improving aggregation consistency, and that it demonstrates outstanding performance in terms of real-time accuracy after aggregation. We answer every question below. In particular, we added:

  • what the one-sided design buys against two-factor sharing (W1, motivation);
  • the Eckart–Young–Mirsky basis of the objective, identifiability, and why only Vᵣ is retained (W1, objective, and Q1);
  • served accuracy, heterogeneous tasks, natural domain shift, a compute-heterogeneity rerun, FedALT and PF2LoRA, and the closest one-sided rules (W2);
  • the statistics, tuning, training stability, and reproduction checks behind every new result (Q2).

All changes are marked in blue in the revised paper, and their locations are given below.


W1 (motivation): The communication overhead is limited

Response to W1 (motivation):

The benefit is personalization, not bandwidth: no aggregation overwrites a client's Bᵢ, and when communication is sparse, this keeps the clients' models accurate. Right after aggregation, PERSIA's clients are +14.26 points above two-factor sharing; with six tasks, two-factor sharing does not improve on local training (79.29 vs 79.49), while PERSIA does (+1.79).

The submitted paper adopted FedSA-LoRA's one-sided design (A captures common knowledge, B adapts to local data; Section 1) and reported 0.76 MiB per round for PERSIA against 1.50 MiB for two-factor sharing (Table 12).

(a) Personalization (W1). As in FedSA-LoRA, no aggregation overwrites a client's Bᵢ, and the server never receives Bᵢ or BᵢAᵢ; omitting Bᵢ also saves 0.75 MiB per client and round.


(b) What this buys (W1), against FedAvg-LoRA (personal head), which shares both factors (3 seeds):

SettingPERSIA − FedAvg-LoRA (personal head), 95% CI
right after the first aggregation of 1,280 steps+14.26 [+11.99, +16.54]
served accuracy, three aggregations of 1,280 steps+5.27 [+4.58, +5.96]
one aggregation of 3,200 steps, retuned (final)+13.60 [+9.09, +18.11]
20 data-scarce clients, two aggregations, retuned (final, MNLI / QNLI / QQP)+5.78 [+3.25, +8.31] / +2.07 [+0.09, +4.04] / +2.59 [−0.02, +5.21]

The General response describes the one-aggregation learning-rate grid.

In the revised paper, Appendix H.7 details what the server learns from the upload, and Appendices H.1 and H.2 give these comparisons.


W1 (objective) and Q1: Validity of the objective; why decompose on the server, and why retain V?

Response to W1 (objective) and Q1:

The objective is valid: it is the personalized aggregation error, which these client messages determine, and it tracks the accuracy lost at aggregation (median Spearman 0.86). The server computes Vᵣ because it depends on all clients, and Vᵣ is the only quantity all clients share.

The submitted paper defined the objective (eq. 2), made it computable from Zᵢ (eqs. 3–4), and proved that the top-r right singular vectors Vᵣ minimize it (Proposition 3.1).

(a) What it optimizes (W1). Since ZᵢᵀZᵢ = ΔWᵢᵀΔWᵢ, minimizing Σᵢ‖ΔWᵢ − Bᵢ′A′‖² over a shared A′ and private Bᵢ′ is a principal-subspace problem on the stack Z̃; the bound is Eckart–Young–Mirsky on Z̃, and each client attains it by rebasing. It is also the aggregation term of Theorem 4.4, which PERSIA minimizes every round.


(b) Why this objective (W1). (RᵢBᵢ, Aᵢ) with any orthogonal Rᵢ sends the same message, so no rule whose messages are functions of Aᵢ and BᵢᵀBᵢ (as PERSIA's and FedSA-LoRA's are) can recover the dense mean or its subspace. These messages do determine Σᵢ ΔWᵢᵀΔWᵢ, hence our objective; factor averaging instead depends on each client's arbitrary coordinates.


(c) It tracks the accuracy lost at an aggregation (W1). Across seven rules on identical client models, the accuracy lost follows the objective (median Spearman 0.86 over 22 runs; pre-specified), and lower error shows in served accuracy (W2).


(d) Why the server, and why V (Q1). The shared subspace depends on all clients, so the server computes it; rebasing needs Bᵢ, so the client does it. Write Z̃ = UΣVᵀ. Block i of UᵣΣᵣ = Z̃Vᵣ is ZᵢVᵣ = OᵢᵀBᵢ⋆, with Oᵢ the polar factor of Bᵢ: client i's rebased factor in a private frame, which the client computes itself as Bᵢ⋆ = ΔWᵢVᵣ. So U carries nothing shared, Σ is absorbed into Bᵢ⋆, and Vᵣ is the only shared quantity on the input space. Any A′ = MVᵣᵀ with Bᵢ′ = Bᵢ⋆M⁻¹ gives the same products; the orthonormal choice makes the rebase a plain product.

In the revised paper, Appendix H.10 states the Eckart–Young–Mirsky basis, identifiability (Remark H.1), and why only Vᵣ is needed, and Appendix H.5 gives the error-versus-accuracy analysis.

AuthorsOpenReview ↗

Response to Reviewer GXNd (2/3)

W2: Overall gains and robustness are limited; PERSIA deteriorates under more heterogeneous clients; few comparisons with recent methods.

Response to W2:

PERSIA's gain is in the model clients use: with sparse synchronization, its served accuracy is +1.43 to +16.53 points above FedSA-LoRA and FedAvg-LoRA (personal head), also under natural domain shift, while final accuracy is on par with FedSA-LoRA. Under compute heterogeneity, it also matches FedSA-LoRA. Among recent methods, PERSIA's mean is above FedALT's at both schedules (clearly so with 5 seeds) and, with sparse communication, matches PF2LoRA at half the uplink.

The submitted paper reported final accuracy on par with the baselines (Table 1: at τ = 1280, PERSIA 91.93, FedAvg-LoRA 91.97, FedSA-LoRA 91.92; at τ = 40, 91.75, 92.65, and 91.80), PERSIA within 1.2 points of FedSA-LoRA across label skew, client count, rank, and participation (Table 3), and a QNLI deficit under compute heterogeneity (Table 11: 88.29 vs 91.33).

(a) Final accuracy (W2). In the tuned rerun of Table 1, PERSIA is on par at τ = 1280 (vs FedSA-LoRA +0.04 [−0.75, +0.84]); with frequent synchronization (τ = 40) two-factor sharing is ahead (−1.72 [−2.08, −1.35]). (3 seeds)


(b) Served accuracy (W2) (the model each client holds, averaged over the run; defined in advance), PERSIA − X (3 seeds):

Settingvs FedSA-LoRAvs FedAvg-LoRA (personal head)
3 aggregations of 1,280 steps, 4 tasks+1.43 [+1.18, +1.67]+5.27 [+4.58, +5.96]
2 aggregations of 1,600 steps, retuned+2.54 [+0.95, +4.13]+11.02 [+9.26, +12.78]
1 aggregation of 3,200 steps, retuned (= final)two added seeds +2.97 [+1.35, +4.58]; three seeds +2.89 [−1.67, +7.45]+13.60 [+9.09, +18.11]

(c) Heterogeneous tasks and natural domain shift (W2). Six GLUE tasks: final +1.79 [+0.19, +3.38] over local and +0.99 [+0.41, +1.57] over FedSA-LoRA, on par with retuned FedAvg-LoRA (personal head) (+1.98 [−0.17, +4.14]); served, ahead of all three. MultiNLI genres (no label skew, learning rates not retuned), three aggregations: served +1.54 [+0.41, +2.66] over FedSA-LoRA and +16.53 [+13.83, +19.24] over FedAvg-LoRA (personal head), at comparable final accuracy (+0.39 [−0.73, +1.51]; +1.59 [−8.45, +11.64]). Our responses to Reviewers N8zs (W1 and Q1) and ELcy (W2) give the tables. (3 seeds)


(d) Compute heterogeneity (W2). In a rerun of Table 11 with one learning rate shared by every method, PERSIA and FedSA-LoRA are on par on the pre-registered validation metric (−0.06 [−0.58, +0.46] severe, −0.01 [−0.91, +0.89] extreme; test +0.69 and −0.90). Our response to Reviewer qd2D (W4 and Q4) gives the reweightings and per-budget results. (3 seeds)


(e) Recent methods (W2). FedALT [1]: with two added seeds (PERSIA and FedALT only), PERSIA − FedALT is +1.33 [+0.03, +2.64] (τ = 40) and +1.14 [+0.24, +2.04] (τ = 1280); with three seeds +1.13 [−1.92, +4.18] and +1.37 [−0.86, +3.61], a higher mean in all four task–schedule cells. PF2LoRA [2] (our implementation): with sparse communication (τ = 1280) the two are on par (91.24 ± 0.03 vs 90.94 ± 0.48; +0.29 [−0.98, +1.56]); with frequent synchronization (τ = 40) PF2LoRA is ahead (92.57 ± 0.42 vs 90.89 ± 0.32; −1.67 [−2.50, −0.85]), at the cost of a second adapter with second-order updates (2.1–2.8× the GPU time of FedAvg-LoRA (personal head), shared GPUs, upper bounds). FedALT took 1.4–2.3× the wall-clock time. Both upload twice PERSIA's values. Closest one-sided rules, on identical client models at the first τ = 1280 aggregation: PERSIA loses 0.05 MNLI points, projector consensus 5.69, FedRot-LoRA 8.53. Our response to Reviewer qd2D (W1 and Q1; W2 and Q2) gives the tables and costs. (3 seeds)

In the revised paper, Appendix H.1 gives the rerun of Table 1 and served accuracy (Tables 21–24), Appendix H.2 the heterogeneous-task and domain-shift results (Tables 25 and 27), Appendix H.3 the closest rules (Table 28), Appendix H.4 FedALT and PF2LoRA (Tables 30 and 33), and Appendix H.5 the rerun (Tables 34 and 35); Section 6 points to each.


[1] Bian, Jieming, et al. "Fedalt: Federated fine-tuning through adaptive local training with rest-of-world lora." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 24. 2026.
[2] Hao, Jie, et al. "Personalized federated fine-tuning for heterogeneous data: An automatic rank learning approach via two-level lora." arXiv preprint arXiv:2503.03920 (2025).
AuthorsOpenReview ↗

Response to Reviewer GXNd (3/3)

Q2: Additional explanation regarding the robustness of the experimental results.

Response to Q2:

Every new comparison reports three seeds and a stated tuning rule, with seed-paired 95% intervals wherever per-seed values exist; the main comparisons tune every method by one procedure with selection on validation and served accuracy, and the domain-shift and LLM analyses were defined before any value was computed. The current code reproduces the submission-era runs we re-ran record for record.

The submitted paper reported the mean ± population standard deviation over three seeds and the per-client majority floor, excluded RTE from the averages (Section 5), and described the rebase guard for collapsed local phases (Appendix B.3).

(a) Statistics (Q2). Final-round unweighted client means, mean ± sample std over three seeds (for PERSIA vs FedALT, and vs FedSA-LoRA with one aggregation, also two added seeds, reported next to the 3-seed result), and seed-paired 95% t-intervals; a lead is claimed only when the whole interval has one sign. We defined and logged served accuracy before computing any value.


(b) Tuning (Q2). The rerun of Table 1 and the fixed-budget, FedALT, rank-sweep, and LLM comparisons tune each compared method, PERSIA included, by one procedure (same search, selection on validation only; test never used for selection); the adapted FedRot-LoRA and projector consensus, and PF2LoRA, got a smaller search. The six-task, data-scarce, 100-client, and compute-heterogeneity runs use one learning rate (2e-4) for every method, with FedAvg-LoRA (personal head) retuned in the six-task and data-scarce settings, where PERSIA led; r = 64 uses the learning rate selected at r = 32.


(c) Training stability (Q2). In the 12 tuned MNLI/QNLI runs of PERSIA, no client ends at its majority class; in the tuned rank sweep, PERSIA and FedSA-LoRA each leave 2 clients there over 18 runs. At a fixed, untuned learning rate a client can stall there under one-sided rules; the rebase guard of Appendix B.3 is designed for this case.


(d) Reproduction (Q2). The current code reproduces the submission-era runs we re-ran record for record (120 of 120 server evaluations of global-protocol PERSIA without error feedback).

In the revised paper, the opening paragraphs of the new Appendix H (protocol, hyperparameter selection, and reliability) give these details.


We thank the reviewer for prompting a clearer explanation of what the one-sided design offers beyond bandwidth savings. Under sparse synchronization, PERSIA exceeds two-factor sharing by 14.26 points immediately after aggregation and, with data-scarce clients, by 5.78 (MNLI) and 2.07 (QNLI) points in final accuracy.

The revision also explains why the server objective is determined by the client messages and why only the shared right subspace is broadcast. It adds served-accuracy, heterogeneity, compute-heterogeneity, FedALT, and PF2LoRA evaluations, together with the tuning procedures and statistical support for the new results.

If these results and clarifications resolve the reviewer's concerns, we would be grateful if the reviewer would reconsider their evaluation. We would be happy to clarify any remaining points.

Reviewer_ELcyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PERSIA for personalized federated LoRA, where the right factor A is globally shared while the left factor B_i remains private. To address the inconsistency caused by LoRA’s non-identifiable factorization, PERSIA uses Gram whitening to construct factorization-invariant representations of client updates, extracts a shared right subspace via SVD, and rebases each client’s private factor accordingly. The authors provide an optimality and convergence analysis and evaluate the method under different federated settings.

Critical strengths
  1. The paper identifies a subtle limitation of one-sided personalized LoRA aggregation. Since LoRA factors are non-identifiable, directly averaging shared A_i factors can be coordinate-dependent even when the corresponding client updates are functionally similar. This is a more fundamental issue than standard parameter-space client drift.

  2. The Gram-whitening construction removes the client-specific metric induced by the private B_i, after which the shared right subspace can be obtained through a principal-subspace problem. The resulting method exactly minimizes the mean reconstruction error among shared rank-r right factors and is invariant to invertible LoRA reparameterizations.

  3. Under long local training intervals, PERSIA substantially reduces the immediate accuracy degradation caused by aggregation. For example, after 1280 local steps and the first aggregation, the reported average accuracy is 91.09% for PERSIA, compared with 86.68% for FedSA-LoRA and 79.53% for two-factor FedAvg-LoRA. This provides direct empirical support for the proposed aggregation objective.

Critical weaknesses
  1. The main empirical advantage appears largely transient and does not consistently improve final personalized performance. PERSIA dramatically improves accuracy immediately after aggregation under long local training horizons, demonstrating that it preserves locally learned functions much better than factor averaging. However, subsequent local training largely repairs the aggregation damage suffered by competing methods. Under the communication-sparse schedule, the final average accuracies of FedAvg-LoRA, FedSA-LoRA, and PERSIA are 91.97%, 91.92%, and 91.93%, respectively. Under the standard schedule, PERSIA is also not better than FedSA-LoRA on average and trails the two-factor baselines. Thus, the paper convincingly establishes better instantaneous aggregation consistency, but it is less clear when this property translates into a practically meaningful improvement such as higher final accuracy, fewer communication rounds, faster convergence to a target accuracy, or improved performance under a fixed communication budget.

  2. The empirical evaluation is relatively narrow for a personalized federated fine-tuning paper. The main personalized evaluation is limited to GLUE classification with RoBERTa-Large and mostly synthetic Dirichlet label heterogeneity. Although the appendix considers additional settings and a global two-factor protocol, stronger evidence on generative tasks and more realistic forms of client/task heterogeneity in the personalized setting would help establish the generality of the proposed subspace-consistency issue and the practical value of PERSIA.

  3. The paper discusses recent personalized federated LoRA approaches such as FDLoRA, PF2LoRA, HiLoRA, and FedALT, but the main personalized experiments focus primarily on FedSA-LoRA and standard global-adapter baselines with personalized heads. I recognize that PERSIA is primarily an aggregation rule and that these methods may use different personalization architectures, so they are not necessarily direct replacements. Nevertheless, integrating PERSIA into, or experimentally comparing against, stronger contemporary personalized LoRA frameworks would better establish whether the proposed aggregation principle provides benefits beyond the specific FedSA-LoRA-style protocol.

  4. The convergence result is stated in terms of factor-space stationarity and assumes full participation, exact SVD, bounded factor iterates, and SGD-style optimization, while the practical implementation includes settings such as partial participation and adaptive optimization. The exact per-round optimality result remains valuable independently of these assumptions, but the convergence theorem provides more limited justification for the full experimental setting.

Questions for authors
  1. Since the largest improvements occur immediately after aggregation but mostly disappear after subsequent local training, can the authors demonstrate a practical advantage in terms of communication rounds or wall-clock time required to reach a fixed target accuracy?
  2. Is there a regime in which improved aggregation preservation leads to consistently better final personalized accuracy, rather than mainly reducing the temporary post-aggregation degradation? For example, does the advantage increase further as communication becomes even sparser or local computation becomes more expensive?
  3. Can PERSIA be incorporated into stronger recent personalized federated LoRA frameworks such as FDLoRA, PF2LoRA, HiLoRA, or FedALT? If so, does the invariant aggregation principle provide additional gains in those architectures?
  4. How sensitive is PERSIA to violations of the shared rank-r subspace assumption? In highly heterogeneous personalized tasks, client-optimal updates may genuinely occupy substantially different right subspaces rather than representing similar functions in different LoRA coordinates. How does PERSIA behave in this regime?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ELcy (1/3)

We thank the reviewer for the careful assessment and for recognizing that PERSIA addresses a more fundamental issue than standard parameter-space client drift, that it exactly minimizes the mean reconstruction error among shared rank-r right factors and is invariant to invertible LoRA reparameterizations, and that its first-aggregation results provide direct empirical support for the proposed aggregation objective. We answer every question below. In particular, we added:

  • served accuracy, 1–10 aggregations at a fixed budget, equal uplink, and data-scarce clients (W1 and Q2);
  • recovery times (Q1);
  • a personalized LLM, heterogeneous tasks, 100 clients, and domain shift (W2);
  • FedALT, PF2LoRA, and PERSIA's rule inside FedALT (W3 and Q3);
  • two extensions of Theorem 4.4 and an SGD check (W4);
  • a test of clients in different right subspaces (Q4).

All changes are marked in blue in the revised paper, and their locations are given below.


W1 and Q2: The advantage is largely transient; is there a regime with consistently better results, e.g. sparser communication or a fixed budget?

Response to W1 and Q2: With six heterogeneous tasks (τ = 40), PERSIA's final accuracy is above local training and FedSA-LoRA (+1.79, +0.99); with sparse communication, its served lead grows. With one aggregation, PERSIA's final (= served) accuracy is +13.60 above FedAvg-LoRA (personal head) and, with two added seeds, +2.97 above FedSA-LoRA; at equal uplink, its final accuracy is +13.65 higher; with data-scarce clients, it ends +5.78 (MNLI) and +2.07 (QNLI) above FedAvg-LoRA (personal head). With frequent aggregation on a single task, sharing both factors is preferable.

The submitted paper reported the gap the reviewer quotes (91.09 vs 86.68 and 79.53) with final accuracy on par (FedAvg-LoRA 91.97, FedSA-LoRA 91.92, PERSIA 91.93; Table 1); the revision keeps these values and adds a rerun with Algorithm 1 as written and every method tuned (Table 21).

(a) With several aggregations, final accuracy is on par (W1). At τ = 1280, PERSIA is level with FedSA-LoRA (+0.04 [−0.75, +0.84]), FedAvg-LoRA (personal head) (+0.02 [−0.47, +0.50]), and local training (+0.23 [−0.32, +0.78]); with frequent aggregation (τ = 40) FedAvg-LoRA (personal head) is ahead (−1.72 [−2.08, −1.35]). (3 seeds)


(b) The transient gap changes the model clients hold (W1). In served accuracy (each client's model averaged over the run, fixed in advance), with three aggregations of 1,280 steps, PERSIA leads FedSA-LoRA by +1.43 [+1.18, +1.67] and FedAvg-LoRA (personal head) by +5.27 [+4.58, +5.96]. Our response to Reviewer N8zs (W2 and Q2) gives the table. (3 seeds)


(c) The lead grows as aggregations become fewer (Q2). Fixed budget of 3,200 local steps, MNLI/QNLI, served PERSIA − FedSA-LoRA / − FedAvg-LoRA (personal head): with 10 aggregations +0.42 [−0.22, +1.07] / −0.47 [−1.30, +0.37], with 3 +1.19 [−3.94, +6.32] / +5.26 [−4.13, +14.64] (τ = 1280 learning rates); with 2 (retuned) +2.54 [+0.95, +4.13] / +11.02 [+9.26, +12.78]; with 1 (retuned; served = final), with two added seeds for both, +2.97 [+1.35, +4.58] (+2.89 [−1.67, +7.45] with three seeds) / +13.60 [+9.09, +18.11]. The General response describes the one-aggregation learning-rate grid; Table 23 of the revised paper gives all four schedules. (3 seeds)


(d) Fixed communication budget (W1). At equal uplink (about 1.5 MiB per client: PERSIA with two aggregations, FedAvg-LoRA (personal head) with one), PERSIA's final accuracy is +13.65 [+8.66, +18.63] higher (+13.61 before the grid extension). (3 seeds)


(e) Data-scarce clients (Q2) (20 clients, 512 examples on average, two aggregations). PERSIA ends +5.78 [+3.25, +8.31] (MNLI), +2.07 [+0.09, +4.04] (QNLI), and +2.59 [−0.02, +5.21] (QQP) above retuned FedAvg-LoRA (personal head); with ten aggregations the methods are comparable. (3 seeds)

In the revised paper, Appendix H.1 gives these results and the grid extension (Tables 21–24).


Q1: A practical advantage in rounds or wall-clock time to reach a fixed target accuracy?

Response to Q1:

Not in steps to a fixed target: the other rules recover within a median of 10 to 30 of 1,280 local steps. PERSIA's practical advantage is the accuracy of the model clients hold in the meantime, and the final accuracy when aggregations are few (W1 and Q2).

The submitted paper observed that local training absorbs most of the other rules' aggregation error within one round (Section 6).

(a) Recovery (Q1). Pre-aggregation accuracy returns after a median of 10 (FedAvg-LoRA (personal head)) and 30 (FedSA-LoRA) steps (one seed, 20 client–task pairs); PERSIA has almost nothing to recover, at a server step of 0.212 s per aggregation (FedSA-LoRA 0.0073 s).

In the revised paper, Appendix H.1 gives the recovery times.

AuthorsOpenReview ↗

Response to Reviewer ELcy (2/3)

W2: The evaluation is narrow; stronger evidence on generative tasks, realistic client and task heterogeneity.

Response to W2:

We added all three: a personalized generative LLM, heterogeneous tasks and many clients, and natural domain shift. On the LLM, PERSIA preserves what clients learn (+7.14 over two-factor averaging after a long phase, +3.19 served accuracy over FedAvg-LoRA). With six heterogeneous tasks and 100 clients, it ends above local training (+1.79; +1.74 and +2.48). Under natural domain shift, its served accuracy leads FedSA-LoRA by +1.54 and FedAvg-LoRA (personal head) by +16.53.

The submitted paper evaluated GLUE under label skew and its variations (Table 3, Appendix C.4), a six-task mixture (Section 6), and a generative model in the global protocol (Appendix E).

(a) Generative LLM, personalized (W2). Qwen2.5-3B-Instruct, eight clients (FedDPA tasks), every method tuned. Final: PERSIA 76.58 ± 1.15, FedSA-LoRA 77.14 ± 0.65, local 76.58 ± 0.65; served, +3.19 [+2.31, +4.06] above FedAvg-LoRA and on par with FedSA-LoRA and local training. After one 1,280-step phase, PERSIA keeps the clients' local accuracy and leads two-factor averaging by +7.14 [+5.89, +8.39]; it is on par with FedSA-LoRA (+0.22 [−0.95, +1.40]). Each client has only 270 training items; a study with larger models and larger client datasets did not fit the rebuttal period. Our response to Reviewer N8zs (Q4) gives the details.


(b) Task heterogeneity and many clients (W2). Six GLUE tasks: final +1.79 [+0.19, +3.38] over local and +0.99 [+0.41, +1.57] over FedSA-LoRA, on par with retuned FedAvg-LoRA (personal head); served, ahead of all three. 100 clients: +1.74 and +2.48 over local, within 0.5 points of FedSA-LoRA; here sharing both factors (FedAvg-LoRA (personal head)) helps more, by about 4 points. Our response to Reviewer N8zs (W1 and Q1) gives the table.


(c) Natural domain shift (W2). Ten MNLI clients, two per MultiNLI genre, no label skew; learning rates from the main comparison (not retuned); analysis fixed in advance. Three aggregations of 1,280 steps, PERSIA − X:

Xservedfinalright after 1st aggregation
FedSA-LoRA+1.54 [+0.41, +2.66]+0.39 [−0.73, +1.51]+3.84 [+1.69, +5.98]
FedAvg-LoRA (personal head)+16.53 [+13.83, +19.24]+1.59 [−8.45, +11.64]+46.88 [+34.53, +59.24]
Local+0.02 [−0.50, +0.55]+0.20 [−0.48, +0.89]−0.22 [−1.02, +0.59]

On the same client models, PERSIA loses −0.02 points at the first aggregation; FedSA-LoRA: 3.83.

In the revised paper, the new Appendix H.6 gives the LLM results (Tables 36–38), and the new Appendix H.2 gives the other three studies (Tables 25–27).


W3 and Q3: Incorporating or comparing with personalized baselines, and gains from the invariant aggregation there.

Response to W3 and Q3:

We ran FedALT and PF2LoRA end-to-end and built PERSIA's rule into FedALT. PERSIA's mean is above FedALT's at both schedules (+1.13, +1.37; two added seeds: +1.33, +1.14) at half its uplink; with sparse communication, it matches PF2LoRA at half the uplink and without second-order computation, while with frequent synchronization, PF2LoRA leads. In FedALT, PERSIA's rule matches FedALT's aggregation without any retuning.

The submitted paper discussed these frameworks in Section 2.

(a) How PERSIA fits (Q3). All four send both factors of their shared adapter, so only PERSIA's two-factor variant applies there; it is on par with FedAvg-LoRA (global protocol, τ = 40: MNLI +0.94 [−0.69, +2.57], QNLI +0.13 [−0.26, +0.52]). PERSIA's contribution is the one-sided case. FDLoRA and HiLoRA are compared by cost only. (3 seeds)


(b) FedALT and PF2LoRA (W3). With two added seeds (PERSIA and FedALT only), PERSIA − FedALT is +1.33 [+0.03, +2.64] (τ = 40) and +1.14 [+0.24, +2.04] (τ = 1280); with three seeds +1.13 [−1.92, +4.18] and +1.37 [−0.86, +3.61], a higher mean in all four task–schedule cells. PF2LoRA (our implementation) is on par with PERSIA at τ = 1280 (+0.29 [−0.98, +1.56]) and ahead at τ = 40 (−1.67 [−2.50, −0.85]), at the cost of Hessian-vector products (2.1–2.8× the GPU time of FedAvg-LoRA (personal head)). Both upload 786,432 values (PERSIA 396,288). Our response to Reviewer qd2D (W2 and Q2) gives the tables and costs. (3 seeds)

(c) PERSIA inside FedALT (Q3). FALT-P builds FedALT's rest-of-world adapter with PERSIA's two-factor rule instead of factor-wise averaging (FedALT's configuration). Final MNLI/QNLI mean: 90.93 ± 0.46 (τ = 40) and 90.17 ± 1.07 (τ = 1280); FALT-P − FedALT: +1.17 [−2.00, +4.34] and +0.31 [−0.07, +0.68], on par (3 seeds). FALT-P's final mean is above FedALT's in all four task–schedule cells, with a clear gain on MNLI at τ = 40 (+0.82 [+0.49, +1.14]). FALT-P thus matches FedALT's final accuracy without retuning.

In the revised paper, the new Appendix H.4 gives these results and costs (Tables 29–33).

AuthorsOpenReview ↗

Response to Reviewer ELcy (3/3)

W4: The convergence result assumes full participation, exact SVD, bounded factor iterates, and SGD, while the implementation uses partial participation and adaptive optimization.

Response to W4:

We extend Theorem 4.4 to partial participation and to an inexact subspace, and an SGD check reproduces PERSIA's first-aggregation preservation and, under the pre-registered tie-tolerant rule, the AdamW ordering of the methods. Adaptive optimization remains outside the theorem, while the per-round optimality of Proposition 3.1 holds for any optimizer.

The submitted paper stated that the result "concerns factor stationarity under full participation and exact SVD, excluding the ridge, and adaptive optimizers used experimentally" (Section 4, Appendix G).

(a) Two extensions (W4). With S of N clients sampled uniformly (absent clients rebase when they next receive A), an analog of part (i) of Theorem 4.4 holds with the optimization term grown by N/S and every rebase displacement in the aggregation term. With an inexact subspace whose expected residual is within (1 + κ) of the bound, part (ii) holds with the floor scaled by √(1 + κ), still independent of τ; a randomized SVD with p ≥ 2 oversampling columns gives κ = r/(p − 1). A rate under partial participation needs a coverage condition.


(b) Optimizer-free part and SGD check (W4). For given incoming factors, PERSIA's aggregation displacement is the least of any shared rank-r right factor (Proposition 3.1; under partial participation, for the participating clients). With SGD and momentum 0.9, at the first τ = 1280 aggregation, PERSIA changes validation accuracy by +0.01 points, against −0.47 (MNLI) and −1.52 (QNLI) for FedSA-LoRA; the ordering matches AdamW in 4 of 4 cells, though PERSIA's mean is lowest in 3. With 100 clients, a client absent for 20 or more rounds loses 3–5% of its update energy when it rebases. (3 seeds)


In the revised paper, Appendix H.10 gives the scope (Table 46), both extensions with proofs (Theorems H.2 and H.3, Corollary H.4), and the SGD check (Table 47).


Q4: Sensitivity when client-optimal updates occupy different right subspaces.

Response to Q4:

PERSIA has the least total loss at every overlap, and in our runs, where client subspaces overlap by 0.61 or more, it has the lowest total and mean loss on every input. When subspaces differ strongly, a low-energy client with a private subspace can lose more than under factor averaging.

The submitted paper covered the two ends: Proposition 3.1 holds for arbitrary client updates, and Corollary 3.2 shows that the error vanishes when all updates share an r-dimensional row space.

(a) Total loss (Q4). Ten synthetic clients with pairwise principal cosine w (5 seeds), energy lost, mean / worst client: PERSIA 0.048 / 0.066 (w = 0.9), 0.203 / 0.294 (w = 0.5), 0.328 / 1.000 (disjoint); FedSA-LoRA and factor averaging + rebasing 0.090, 0.450, and 0.900 for every client. No rule had a smaller total in any of the 3,420 instances.


(b) Individual clients can lose more (Q4). A low-energy client with a private subspace is dropped first: at w = 0.9, a client with 0.1× the energy loses 0.162 under PERSIA against 0.090 under factor averaging; with disjoint subspaces, some client loses its whole update.


(c) Where our runs sit (Q4). Effective overlap is 0.61–0.66 (SST-2: 0.91) after the first τ = 1280 phase and 0.92 or more later. PERSIA has the lowest total and mean loss on every input; its worst-served client (the lowest-energy one in 19 of 24 inputs) loses 0.39–0.40 at the first τ = 1280 aggregation (client mean 0.12–0.13).


In the revised paper, Appendix H.10 gives the controlled test (Table 45) and the run analysis.


We thank the reviewer for raising the question of whether better preservation at aggregation can translate into better final personalized accuracy. The added experiments identify such a regime: with six heterogeneous tasks, PERSIA's final accuracy exceeds local training, and FedSA-LoRA (+1.79 [+0.19, +3.38], +0.99 [+0.41, +1.57]), and its served accuracy leads all three baselines.

The revision also adds an LLM, heterogeneous tasks, 100 clients, domain shift, FedALT, PF2LoRA, and an extension of Theorem 4.4.

If these results resolve the reviewer's concerns, we would be grateful if the reviewer would reconsider their evaluation. We would be happy to clarify any remaining points.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer ELcy,

Thank you for your time and careful assessment of our work. Following your point that the advantage could be transient, we now report served accuracy, the model each client actually holds over the run: over three aggregations, PERSIA is +1.43 over FedSA-LoRA and +5.27 over FedAvg-LoRA with a personal head.

We also added heterogeneous settings where the advantage reaches final accuracy: +1.79 over local training and +0.99 over FedSA-LoRA with six tasks, and +1.74 / +2.48 over local training with 100 clients.

As the discussion period ends soon, we would like to check whether our responses address your concerns, and we would be happy to clarify anything further.

Best, Paper43965 Authors

Reviewer_EpMBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the severe coordinate misalignment problem in personalized federated LoRA fine-tuning, where naive averaging of shared right factors (while keeping left factors strictly local) leads to catastrophic knowledge cancellation due to arbitrary local basis transformations. To solve this, the authors propose PERSIA, an elegant aggregation algorithm that first applies client-side Gram whitening to encode local geometric metrics without exposing private left factors, then extracts a consistent shared right subspace via server-side truncated SVD, and finally rebases clients' local factors into this new global coordinate system. Theoretically, PERSIA is proven to exactly minimize the mean client reconstruction error while being mathematically invariant to equivalent LoRA factorizations, supported by a convergence bound that remains robust even with long local training intervals. Extensive experiments on the GLUE benchmark demonstrate that PERSIA successfully prevents post-aggregation performance collapse—preserving local knowledge perfectly even under sparse communication—and achieves final accuracy comparable to full-adapter sharing methods while cutting the uplink communication cost in half.

Critical strengths
  • The paper accurately identifies a critical bottleneck in personalized federated LoRA (where the right factor AA is shared and the left factor BB is kept local): the coordinate misalignment problem caused by the non-identifiability of LoRA (BA=BTT−1ABA = BTT^{-1}A). Addressing this low-rank aggregation inconsistency is highly relevant and valuable to the federated learning and PEFT communities.
  • The proposed PERSIA algorithm is intuitively clever and mathematically elegant. Utilizing Gram whitening (inspired by polar decomposition, Zi=(Bi⊤Bi)1/2AiZ_i = (B_i^\top B_i)^{1/2} A_i) successfully encodes the local geometric distortion without explicitly transmitting the private matrix BiB_i. The subsequent server-side truncated SVD and client-side rebasing form a unified and principled framework.
  • The theoretical contributions are exceptionally solid. Proposition 3.1 rigorously proves that PERSIA minimizes the exact mean client reconstruction error and is invariant to equivalent LoRA factorizations (resolving the GL(r)GL(r) symmetry ambiguity). Furthermore, the convergence bound (Theorem 4.4) demonstrating that the stationary floor does not grow with the local steps (τ\tau) perfectly justifies its success in sparse communication regimes.
  • The experimental design is highly insightful, particularly the evaluation of model accuracy immediately after aggregation. The results clearly demonstrate that baseline methods suffer catastrophic knowledge cancellation (dropping up to 10 points), whereas PERSIA almost perfectly preserves the learned local knowledge, matching two-factor sharing performance while cutting the uplink communication cost in half.
Critical weaknesses
  • While PERSIA successfully avoids transmitting the raw BiB_i matrix (maintaining the architecture of PFL), uploading ZiZ_i inherently reveals the local Gram matrix Bi⊤BiB_i^\top B_i (an r×rr \times r matrix per layer) to the server. The paper explicitly states that it adds no privacy guarantee beyond standard FL. However, in PFL, keeping BiB_i local is often motivated by privacy preservation, and leaking the correlation of BiB_i might compromise this objective.
  • The server is required to perform a truncated SVD on the stacked matrix Z~\tilde{Z} of size (Nr)×d(Nr) \times d. While the computational cost O(Ndr2)O(Ndr^2) is negligible for N=10N=10 clients (as used in the experiments), it could become a significant memory and computational bottleneck in realistic cross-device FL settings where the number of clients NN could reach thousands or tens of thousands.
  • The experiments predominantly use a small rank (r=8r=8). It is unclear whether the geometric alignment remains as effective, and whether the server-side SVD overhead remains acceptable, when a larger rank (e.g., r=32,64r=32, 64, or 128128) is required for more complex tasks.
Questions for authors
  • Could you elaborate on the potential privacy risks associated with exposing the Gram matrix Bi⊤BiB_i^\top B_i to the server? Would it be possible to integrate privacy-preserving mechanisms (such as adding DP noise to ZiZ_i) without destroying the metric-awareness that makes PERSIA successful?
  • How does the server-side SVD operation scale when NN is significantly larger (e.g., N=1,000N=1,000 or 10,00010,000)? Have you considered using sketching algorithms, recursive SVD, or other streaming approximations to avoid explicitly forming and storing the large (Nr)×d(Nr) \times d matrix in memory?
  • How does PERSIA perform when the LoRA rank is much larger (e.g., r=64r=64)? Does the performance gap between PERSIA and the baseline factor-averaging methods remain as significant, and does the computational overhead become a limiting factor?
3 comments
AuthorsOpenReview ↗

Response to Reviewer EpMB (1/2)

We thank the reviewer for the positive assessment and for recognizing PERSIA as intuitively clever and mathematically elegant, its theoretical contributions as exceptionally solid, and the experimental design as highly insightful, particularly the evaluation of model accuracy immediately after aggregation. We answer every question below. In particular, we added:

  • what the server learns from Zᵢ, compatibility with secure aggregation and DP-SGD, a noise experiment, and a formal client-level DP analysis of the uploads (W1 and Q1);
  • an exact streamed server step with timings up to N = 10,000, and a test of randomized sketching (W2 and Q2);
  • first-aggregation fidelity at r = 64, final accuracy across ranks, and the overhead at r = 64 and r = 128 (W3 and Q3).

All changes are marked in blue in the revised paper, and their locations are given below.


W1 and Q1: Privacy risk of revealing BᵢᵀBᵢ, and DP noise on Zᵢ.

Response to W1 and Q1:

The server learns each client's singular values and right singular vectors, not its output directions. PERSIA is compatible with secure aggregation and with record-level DP-SGD at FedSA-LoRA's privacy cost, and its aggregation error stays below FedSA-LoRA's under noise on the uploads up to σ = 0.5.

The submitted paper stated that Zᵢ reveals ZᵢᵀZᵢ = ΔWᵢᵀΔWᵢ and that PERSIA adds no privacy guarantee beyond standard FL (Section 3.1).

(a) What the server learns (Q1). The output directions stay hidden: (QBᵢ, Aᵢ) gives the same upload for any orthogonal Q.


(b) Nested disclosure (W1). Each client whitens locally and uploads only Zᵢ = (BᵢᵀBᵢ)^(1/2)Aᵢ, the same size as FedSA-LoRA’s Aᵢ; Bᵢ never leaves the client. From Zᵢ the server learns ZᵢᵀZᵢ = ΔWᵢᵀΔWᵢ, i.e., the singular values and right singular vectors of the update, whereas two-factor sharing reveals ΔWᵢ itself. Keeping Bᵢ local also serves personalization: no aggregation overwrites a client’s Bᵢ.


(c) Secure aggregation and DP-SGD (Q1). The server needs only Σᵢ ZᵢᵀZᵢ: a secure sum of symmetric d × d matrices (about 64× the upload at d = 1,024), or block power iteration with Aᵢ-sized messages. Under record-level DP-SGD every upload, the Gram included, is post-processing, so PERSIA pays exactly FedSA-LoRA's privacy cost, and Proposition 3.1 holds for the noisy factors. Clipping Zᵢ clips ΔWᵢ; i.i.d. Gaussian noise leaves the target subspace unchanged in expectation and the rebase exact, so only the shared subspace degrades.


(d) Noise on the uploads (Q1). We added Gaussian noise of relative size σ to each Zᵢ of saved client states (first aggregation at τ = 1280 on four tasks; aggregations 1–2 at τ = 320 on MNLI/QNLI; 20 draws; FedSA-LoRA: same noise on Aᵢ). Relative reconstruction error on MNLI, τ = 1280: PERSIA 0.103 (σ = 0) and 0.131 (σ = 0.5), FedSA-LoRA 0.451 and 0.463. PERSIA's error stays below FedSA-LoRA's at every σ ≤ 0.5, on every input and in every draw: the metric-awareness survives the noise. The subspace turns by 1.3° (σ = 0.05) and 12.9° (σ = 0.5). This probe has no clipping or accounting; (e) gives formal DP.


(e) Formal client-level DP on the uploads (Q1). Each client clips Zᵢ (FedSA-LoRA: Aᵢ) to the median norm and adds Gaussian noise; 3 aggregations × 48 matrices compose exactly (δ = 1e-5). With only 10 clients, the noise that client-level DP requires at ε ≤ 16 overwhelms the uploads of either rule (MNLI right after aggregation: PERSIA 89.06 → 57.9–59.8, FedSA-LoRA 80.71 → 35.3–53.5). With noise added once to the secure sum Σᵢ ZᵢᵀZᵢ, PERSIA recovers the subspace when clients are many (extrapolated, first aggregation: 100,000 clients, every ε ≥ 1; MNLI 88.84 vs 80.02 for FedSA-LoRA at ε = 1). With few clients, record-level DP-SGD is the practical route, and PERSIA supports it at no extra privacy cost.

In the revised paper, the new Appendix H.7 states what Zᵢ reveals and gives the secure-aggregation and DP-SGD discussion, the noise experiment, and the client-level DP analysis (Table 41).

AuthorsOpenReview ↗

Response to Reviewer EpMB (2/2)

W2 and Q2: Server SVD at N = 1,000 or 10,000, and sketching or streaming approximations.

Response to W2 and Q2:

The exact server step can be streamed in O(d²) memory, independent of N, and takes 17.5 ms per module at N = 10,000; a randomized sketch needs oversampling.

The submitted paper analyzed the server cost up to N = 500 (Section 3; Appendix D.2), proposed a randomized truncated SVD that forms no d × d matrix, and timed N = 10 (Appendix D.3).

(a) Exact streaming (Q2). The server adds each ZᵢᵀZᵢ = ΔWᵢᵀΔWᵢ as it arrives, keeps one d × d matrix, and takes its top-r eigenvectors. Memory is O(d²), independent of N, and the result is exact. Measured times per module (shared GPUs, so upper bounds):

r, dNthin SVD of the stackstreamed exactextra memory, fp64 (thin SVD vs streamed)
8, 1,02410,000225 ms17.5 ms1.93 GiB vs 330 MiB
64, 4,09610,000does not fit (19.5 GiB stack)2.0 sn/a

(b) Sketching (Q2). With q = r columns, a randomized SVD misses the top-r subspace of synthetic stacks (median principal-angle sine 0.22–0.72, moderately decaying spectrum); oversampling to q = r + 8 fixes this, except on a slowly decaying spectrum (0.03–0.29). On the whitened stacks of our runs, q = 2r matches the exact subspace (median sine ≤ 2e-4). For large N, we recommend the exact streamed route; our runs use an exact thin SVD. At N = 10, the full server step takes 0.212 s per aggregation on CPU (FedSA-LoRA 0.0073 s).

In the revised paper, the Scalability paragraph of Section 3 adds oversampling and the exact streamed variant for very large N; the new Appendix H.8 gives the timings and the sketching test.


W3 and Q3: Performance and overhead at larger rank (e.g. r = 64).

Response to W3 and Q3:

The geometric alignment stays effective: at r = 64, PERSIA still keeps local accuracy right after aggregation, while two-factor averaging (FedAvg-LoRA (personal head)) loses 15.10 points more. The overhead stays small: the streamed server step takes 25.5 ms per module at r = 64 (N = 10) and 118.4 ms at r = 128 even with N = 10,000.

The submitted paper reported final accuracy at r = 4 and r = 32 (Table 3, τ = 40; at r = 32, PERSIA 89.32 vs FedSA-LoRA 88.47 on MNLI and 91.32 vs 92.43 on QNLI).

(a) Fidelity at r = 64 (Q3). MNLI/QNLI, τ = 1280, every rule applied to the same incoming client models (learning rate selected at r = 32). Right after the first aggregation PERSIA stays at local accuracy (PERSIA − local −0.01 [−0.11, +0.08]). FedAvg-LoRA (personal head) loses 15.10 [+11.10, +19.10] points more than PERSIA. FedSA-LoRA loses 1.28 [−0.60, +3.15] points more, comparable at this rank (6.50 [+1.92, +11.07] at r = 8). (3 seeds)


(b) Final accuracy across ranks (Q3) (learning rate tuned per method and rank, τ = 40, validation). PERSIA is on par with FedSA-LoRA at r = 4, 16, and 32 (r = 32: +0.11 on MNLI, +0.21 on QNLI). (3 seeds)


(c) Overhead (W3). At d = 1,024 the streamed server step takes 25.5 ms per module at r = 64 and N = 10, and, at r = 128, 11.3, 21.5, and 118.4 ms at N = 10, 1,000, and 10,000 (GPU, fp32; measured separately on shared GPUs, so upper bounds and not comparable across ranks). The client adds only an r × r Gram root and a rebase, O(dr²).

In the revised paper, the new Appendix H.9 gives the rank results (Tables 42–44), and Appendix H.8 the timings at r = 128.


We thank the reviewer again for the positive assessment and for the questions about privacy and scalability. Appendix H.7 now specifies exactly what the client upload reveals, explains compatibility with secure aggregation and record-level DP-SGD at FedSA-LoRA's privacy cost, and evaluates aggregation error under upload noise.

The scalability question also prompted an exact streamed server implementation in Appendix H.8, which processes 10,000 clients in milliseconds at dimension 1,024. The additional rank-64 evaluation shows that PERSIA preserves local accuracy immediately after aggregation with small overhead.

If these additions address the reviewer's concerns, we would be grateful if the reviewer would reconsider their evaluation and support the paper in the discussion. We welcome any further questions.

AuthorsOpenReview ↗

Thank you for the positive assessment

Dear Reviewer EpMB,

Thank you for your time and positive assessment. In response to your questions, we showed that PERSIA is compatible with secure aggregation and record-level DP-SGD at FedSA-LoRA's privacy cost, added an exact streamed server step (17.5 ms per module at N = 10,000), and found that at rank 64, PERSIA maintains local accuracy right after aggregation. We would be happy to clarify anything further before the discussion closes.

Best, Paper43965 Authors

Reviewer_qd2DOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PERSIA to address aggregation inconsistency in personalised federated LoRA, where the right factor (A) is shared and the left factor (B) remains local. Each client transforms (A) using the square root of the Gram matrix of its local (B). The server then extracts a shared right subspace through truncated SVD of the stacked transformed factors, and clients rebase their private (B) onto the new basis. The authors prove that the core rule minimises average reconstruction error for a fixed shared rank, with the selected subspace invariant to equivalent LoRA factorisations. Experiments show that PERSIA reduces accuracy loss at the first aggregation after long local training. However, after further rounds in the sparse communication setting, its final accuracy is similar to that of the baselines.

Critical strengths
  1. The problem addressed in this paper is clearly defined: how to reasonably aggregate the shared factor (A) when the server cannot access (B). The method does not require uploading the full adapter, involves low communication overhead, and features a relatively straightforward core workflow.
  2. The paper provides a proof of the method’s optimality for the reconstruction objective and uses ablation studies to distinguish between the roles of whitening and rebasing. The experiments also accurately demonstrate the method’s limitations, including a lack of significant improvement in final accuracy and a decline in performance under conditions of strong computational heterogeneity.
Critical weaknesses
  1. The method offers limited innovation. This paper primarily employs established tools such as the Gram transform, truncated SVD and rebasing for the aggregation of (A) alone; overall, the level of innovation is limited. Whilst the optimality of the reconstruction justifies the method’s rationale, it is insufficient to demonstrate novelty; the authors should clarify what new technical contributions the method offers compared to existing factor alignment and subspace aggregation methods.

  2. The paper contains too few comparisons with the latest representative personalised federated LoRA methods, such as PF2LoRA and FedALT, which are cited but not experimentally evaluated. It is recommended that relevant baselines be included, with comparisons made across accuracy, the number of trainable parameters, and computational and communication overhead.

3.The practical benefits are not yet sufficiently clear. PERSIA’s main advantage lies in reducing the performance loss during the first aggregation, but post-aggregation accuracy is close to that of locally trained models, and after three rounds in the sparse communication setting, it is essentially on a par with other methods. It should be verified whether, under the same communication volume or training time, the method can deliver significant practical benefits when communication is limited and the model is used immediately after aggregation.

  1. Proposition 3.1 only guarantees that the weight reconstruction error is minimised; it does not guarantee the best prediction performance. Under severe computational heterogeneity, PERSIA lags significantly behind FedSA-LoRA on QNLI (88.29% vs. 91.33%), highlighting a practical limitation of the method. Please evaluate the effects of normalisation or reweighting, analyse the relationship between reconstruction error and accuracy, and report performance across client computational budgets to assess whether minimising the average error comes at the expense of some clients’ performance.
Questions for authors
  1. What is the main technical contribution beyond combining Gram whitening, truncated SVD, and rebasing? How does PERSIA differ from existing factor alignment and subspace aggregation methods?

  2. How does PERSIA compare with PF2LoRA and FedALT in accuracy, the number of trainable parameters, and computational and communication costs?

  3. With the same total communication or training-time budget, does PERSIA offer a clear advantage when models are used after only one or two aggregations?

  4. Can normalisation or reweighting improve performance under severe computational heterogeneity? How do reconstruction error and accuracy vary across clients with different computational budgets?

  5. What value of ε is used? How is the error feedback buffer stored, updated and compressed, and is its size fixed?

6 comments
AuthorsOpenReview ↗

Response to Reviewer qd2D (1/3)

We thank the reviewer for the careful assessment and for recognizing that PERSIA addresses a problem that is clearly defined, that it does not require uploading the full adapter, involves low communication overhead, and that the paper provides a proof of the method’s optimality for the reconstruction objective. We answer every question below. In particular, we added:

  • the Eckart–Young–Mirsky basis, identifiability, and the closest alignment and subspace rules on identical client models (W1 and Q1);
  • FedALT and PF2LoRA run end-to-end, with parameters, uplink, and compute (W2 and Q2);
  • served accuracy at a fixed budget of local steps and an equal-uplink comparison (W3 and Q3);
  • a compute-heterogeneity rerun with both reweightings, per-budget results, and error versus accuracy (W4 and Q4);
  • the value of ε and an exact description of the error-feedback buffer (Q5).

All changes are marked in blue in the revised paper, and we list their locations below.


W1 and Q1: Contribution beyond the established tools, and difference from factor-alignment and subspace methods.

Response to W1 and Q1:

The tools are classical; what is new is that the server solves the joint problem exactly from right-factor messages, and that no rule seeing only these messages can recover the dense mean (identifiability). On identical client models, PERSIA loses 0.05 MNLI points at an aggregation, against 5.69 (projector consensus) and 8.53 (FedRot-LoRA).

The submitted paper proved optimality and invariance (Proposition 3.1) and noted that FedRot-LoRA and GLoRA require both factors at the server (Section 2).

(a) Exact reduction (Q1). The bound of Proposition 3.1 is Eckart–Young–Mirsky on the whitened stack. What is new is that the joint problem (shared A′, private Bᵢ′) is posed on Bᵢ and ΔWᵢ, which the server never sees, yet ZᵢᵀZᵢ = ΔWᵢᵀΔWᵢ lets the server solve it exactly from right-factor-sized messages (the r × r Gram adds 0.78% to the upload). Each client attains its optimal Bᵢ′ = ΔWᵢVᵣ by rebasing.


(b) Identifiability (Q1). Replacing Bᵢ by RᵢBᵢ, with an orthogonal Rᵢ per client, leaves every message unchanged but changes ΔWᵢ. So no rule whose client messages are functions of Aᵢ and BᵢᵀBᵢ (as PERSIA's and FedSA-LoRA's are) can recover the dense mean or its right subspace. These messages do determine Σᵢ ΔWᵢᵀΔWᵢ, which is PERSIA's objective.


cc How the objective differs (W1). PERSIA takes the top-r eigenvectors of Σᵢ ΔWᵢᵀΔWᵢ: energy-weighted and invariant to refactorization. GLoRA's projector consensus, moved to the row space, uses Σᵢ Πᵢ: invariant, but blind to magnitude. The unwhitened SVD uses Σᵢ AᵢᵀAᵢ, which is not invariant. FedRot-LoRA corrects only rotations.


(d) Same client models, first aggregation after 1,280 local steps (Q1). Accuracy lost, in points (FedRot-LoRA and projector consensus adapted to our protocol; 3 seeds, means):

TaskPERSIAw/o whiteningfactor avg. + rebasingprojector consensusFedRot-LoRAFedSA-LoRA
MNLI0.051.073.005.698.538.61
QNLI−0.27−0.540.721.694.174.11

On MNLI, rebasing cuts the loss from 8.61 to 3.00, and whitening cuts it from 1.07 to 0.05.


(e) Role of whitening (W1). Whitening is what makes the server's subspace independent of how each client factorizes its update, and what makes the aggregation exactly optimal (Proposition 3.1). On identical client models, it removes almost all of the remaining loss (MNLI points lost 1.07 → 0.05, (d)).

In the revised paper, Appendix H.10 gives the Eckart–Young–Mirsky basis and identifiability (Remark H.1), and the new Appendix H.3 gives the closest rules (Table 28).

AuthorsOpenReview ↗

Response to Reviewer qd2D (2/3)

W2 and Q2: Comparison with PF2LoRA and FedALT in accuracy, trainable parameters, and computation and communication costs.

Response to W2 and Q2:

PERSIA trains as many parameters as FedSA-LoRA and uploads half as much as FedALT and PF2LoRA. Its mean is above FedALT's at both schedules (+1.13 and +1.37 with three seeds), clearly so with two added seeds (+1.33 and +1.14). With sparse communication, it matches PF2LoRA at half the uplink and without second-order computation; with frequent synchronization, PF2LoRA is ahead.

The submitted paper cited both and noted that their shared component is a complete adapter aggregated by the server, so an aggregation rule such as PERSIA is complementary (Section 2). All methods share clients, splits, rank, schedules, and a local head.

(a) Cost per client (Q2) (RoBERTa-Large, r = 8, heads excluded; server step with 10 clients on CPU, relative to FedAvg-LoRA (personal head) timed in the same session; PERSIA: 0.212 s):

Methodtrainable LoRAuplink per round (bf16)extra client workserver step
FedSA-LoRA0.79M0.75 MiBnone0.5×
PERSIA0.79M0.76 MiBGram root + rebase, O(dr²)14×
FedALT0.79M + mixer 0.10M (+0.79M frozen)1.50 MiBsecond adapter + mixer in every forward1.0×
PF2LoRA (client rank 2)0.98M1.50 MiBHessian-vector products (4.65× FLOPs in its paper; in our runs 2.1–2.8× the GPU time of FedAvg-LoRA (personal head))1.2×

(b) Accuracy (Q2). Final test accuracy, MNLI/QNLI mean (3 seeds):

Methodτ = 40τ = 1280
PERSIA90.89 ± 0.3291.24 ± 0.03
FedALT89.76 ± 0.9389.86 ± 0.93
PF2LoRA92.57 ± 0.4290.94 ± 0.48

FedALT (same tuning procedure as our other baselines): PERSIA has the higher mean in all four task–schedule cells, and with two added seeds (PERSIA and FedALT only) its lead is clear at both schedules, +1.33 [+0.03, +2.64] and +1.14 [+0.24, +2.04] (added seeds; 3 seeds: +1.13 [−1.92, +4.18], +1.37 [−0.86, +3.61]). FedALT runs took 1.4–2.3× the wall-clock time of FedAvg-LoRA (personal head).

PF2LoRA: no code is public, so we implemented it from its paper (a shared rank-8 adapter and a private rank-2 adapter per client, trained by its bilevel update; Hessian-vector products checked against finite differences) and synchronized it every τ steps, like every other method. With sparse communication (τ = 1280), the setting PERSIA targets, the two are on par (+0.29 [−0.98, +1.56]), while PERSIA uploads half as much and needs no Hessian-vector products. With frequent synchronization (τ = 40), PF2LoRA is ahead (−1.67 [−2.50, −0.85]) at this higher cost.

In the revised paper, the new Appendix H.4 gives the cost table (Table 29), FedALT (Table 30), and PF2LoRA (Tables 32 and 33).


W3 and Q3: Advantage at the same communication or training-time budget, with one or two aggregations.

Response to W3 and Q3:

Yes. At the same 3,200 local steps, PERSIA's served accuracy is +11.02 and +13.60 points above FedAvg-LoRA (personal head) with two and one aggregations, and at the same uplink, its final accuracy is +13.65 higher; against FedSA-LoRA, the lead is +2.54 with two aggregations and +2.97 with one (two added seeds; +2.89 with three).

The submitted paper fixed the local work at τT = 3,200 steps (Section 5); the revision measures one and two aggregations and equal uplink at that budget.

(a) Same training time: 3,200 local steps (Q3). Local training is identical across methods, so equal time means equal local steps. Served accuracy: each client's test accuracy at every logged evaluation, averaged over the run, then over clients (defined in advance). MNLI/QNLI, learning rates retuned (3 seeds):

AggregationsPERSIAFedSA-LoRAFedAvg-LoRA (personal head)Local
290.35 ± 0.1887.81 ± 0.6079.33 ± 0.7990.26 ± 0.56
1 (served = final)90.45 ± 0.5387.56 ± 1.3276.85 ± 1.8188.79 ± 1.80

PERSIA − FedAvg-LoRA (personal head): +11.02 [+9.26, +12.78] (two), +13.60 [+9.09, +18.11] (one). PERSIA − FedSA-LoRA: +2.54 [+0.95, +4.13] (two); with one, +2.97 [+1.35, +4.58] with seeds 3–4 added for both (added seeds; 3 seeds: +2.89 [−1.67, +7.45]). The one-aggregation learning-rate grid was extended below its smallest value; our General response gives the details.


(b) Same total communication (W3). PERSIA with two aggregations and FedAvg-LoRA (personal head) with one each upload about 1.5 MiB per client; PERSIA's final accuracy is +13.65 [+8.66, +18.63] higher. With three aggregations of 1,280 steps, PERSIA's served accuracy is +1.43 above FedSA-LoRA and +5.27 above FedAvg-LoRA (personal head); our response to Reviewer N8zs (W2 and Q2) gives that table. (3 seeds)

In the revised paper, Appendix H.1 gives these results and the grid extension (Tables 22–24).

AuthorsOpenReview ↗

Response to Reviewer qd2D (3/3)

W4 and Q4: Reconstruction optimality versus prediction, the QNLI deficit under compute heterogeneity (88.29 vs 91.33), reweighting, and per-budget results.

Response to W4 and Q4:

Under the same heterogeneous budgets, PERSIA matches FedSA-LoRA. In a rerun with Algorithm 1 and one learning rate for every method, the difference is −0.06 points (severe) and −0.01 (extreme), so no normalization or reweighting is needed. Across budgets, the clients that lose accuracy are those with the fewest local steps under both methods, and the accuracy lost at aggregation follows the reconstruction error (Spearman 0.86).

The Table 11 runs of the submitted paper (PERSIA 88.29 vs FedSA-LoRA 91.33 on QNLI, severe budgets) used the implementation of Appendix B.3. The revision adds a rerun with Algorithm 1 for every method, point by point:

(a) Rerun (W4). QNLI, T = 40, Algorithm 1, learning rate 2e-4 for all, validation fixed as the metric in advance. PERSIA − FedSA-LoRA is −0.06 [−0.58, +0.46] (severe) and −0.01 [−0.91, +0.89] (extreme); on test, +0.69 [+0.24, +1.14] and −0.90 [−1.14, −0.66]. (3 seeds)


(b) Normalization and reweighting (Q4). Change vs uniform weights (mild / severe / extreme): block-normalized −1.49 / −0.10 / +0.19; inverse-√τ −0.55 / −0.28 / +0.15. Uniform weights, as in Algorithm 1, are as good as or better than the others in all but the extreme setting, where the gains stay below 0.2 points. (3 seeds)


(c) Error vs accuracy (W4). Proposition 3.1 bounds what one aggregation keeps. To test this, we applied seven aggregation methods (PERSIA, PERSIA without whitening, FedSA-LoRA, factor averaging with rebasing, projector consensus, FedRot-LoRA, and two-factor averaging) to the same client models at the first aggregation of 24 training runs and measured the accuracy each method loses: aggregation methods with lower reconstruction error lose less (median Spearman correlation 0.86; threshold 0.7 fixed in advance; Appendix H.3, Table 28).


(d) Per budget (Q4) (2–4 clients per group; validation; change vs a uniform budget). Extreme: the 5-step clients drop by 3.81 [+0.15, +7.47] under PERSIA and 3.16 [−0.38, +6.69] under FedSA-LoRA. Severe: the 10-step clients drop by 2.80 [−1.51, +7.11] under PERSIA and 0.54 [−1.49, +2.57] under FedSA-LoRA. These clients have the smallest rebase residual, so their loss does not stem from how the aggregation reconstructs their updates. (3 seeds)

In the revised paper, the new Appendix H.5 gives these results (Tables 34 and 35); Section 6, Appendix C.4, and Remark F.6 are updated accordingly, and the Table 11 caption states its configuration.


Q5: The value of ε, and how the error-feedback buffer is stored, updated, and compressed.

Response to Q5:

ε = 1e-12. The buffer is a fixed-size server-side matrix, never sent or compressed.

The submitted paper defined the ridge (BᵢᵀBᵢ + εIᵣ)^(1/2) (Section 3.1) and the buffer (Appendix B.3; removing it changes final accuracy at τ = 40 by at most 0.44 points, Table 4).

(a) ε (Q5) is raised by a factor of 1e2 to 1e6 only if the double-precision eigendecomposition fails; rebasing uses a 1e-4 ridge.


(b) The buffer (Q5), an implementation addition outside Algorithm 1, is one r × d matrix per adapted weight (1.5 MiB in fp32 for all 48). Each round, it is overwritten with E = S − SVᵣVᵣᵀ, S = Σᵢ BᵢᵀBᵢAᵢ, and appended to the next stack as r rows with weight c = 0.5·min(1, ‖E‖/‖S‖). Removing it changes accuracy at τ = 1280 by +0.36 [−0.34, +1.05] (MNLI) and +0.12 [−0.67, +0.91] (QNLI). (3 seeds)

In the revised paper, Appendix B.3 defines ε and describes the buffer in detail.


We thank the reviewer for the constructive feedback and for proposing the matched-budget evaluation. The revision clarifies the contribution of the exact reduction, compares the closest aggregation rules on identical client models, reports FedALT and PF2LoRA with their costs, and adds served-accuracy and compute-heterogeneity evaluations, including reweighting and per-budget results. It also clarifies the ε term and buffer handling.

The reviewer's practical test now has a direct answer: with one aggregation and the same 3,200 local steps, PERSIA improves final accuracy over FedAvg-LoRA with a personal head by 13.60 points; at matched uplink, the improvement is 13.65 points.

If these clarifications and results resolve the reviewer's concerns, we would be grateful if the reviewer would reconsider their evaluation. If a specific concern remains decisive, we would appreciate the opportunity to address it during the discussion.

Reviewer_N8zsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PERSIA, a subspace-based aggregation method for personalized federated LoRA, addressing the inconsistency caused by averaging non-identifiable low-rank factors. PERSIA uses Gram whitening to capture the geometry of client updates, truncated SVD to identify a shared right subspace, and local rebasing to preserve personalized updates without sharing private left factors. The paper provides theoretical guarantees on reconstruction optimality and factor-level convergence. Experiments on GLUE with RoBERTa-Large demonstrate that PERSIA substantially reduces performance degradation immediately after aggregation, particularly under sparse communication, while achieving final accuracy comparable to existing federated baselines with low communication overhead.

Critical strengths
  1. I find the motivation clear. The paper identifies a meaningful aggregation inconsistency in personalized federated LoRA, where averaging non-identifiable factors may fail to preserve locally learned updates. PERSIA addresses this issue through a principled subspace-based approach, with a clear connection between the identified problem, the proposed solution, and its theoretical justification. I particularly appreciate the reconstruction optimality guarantee, which strengthens the methodological contribution.

  2. I appreciate that the experiments directly evaluate how well aggregation preserves local knowledge. The strong results under sparse communication, together with component ablations and comparisons across communication schedules, provide solid support for the proposed method.

Critical weaknesses
  1. PERSIA effectively preserves local model performance after aggregation, which is an important result. However, its final accuracy is only slightly higher than local-only training and remains comparable to existing federated baselines. This raises a question about whether the proposed aggregation enables meaningful knowledge sharing across clients beyond preventing performance degradation.

  2. PERSIA shows substantial improvements immediately after aggregation with long local training intervals. However, these advantages largely diminish after subsequent training rounds, with similar final performance across different aggregation methods. The practical significance of this improvement would be more convincing if the authors could better establish when preserving immediate post-aggregation performance is particularly valuable.

Questions for authors
  1. While PERSIA preserves local model performance after aggregation, its final accuracy is only slightly better than local-only training. Could the authors clarify what evidence demonstrates meaningful knowledge sharing across clients beyond mitigating aggregation-induced degradation?

  2. The strongest improvements appear immediately after aggregation, while different methods achieve similar accuracy after further local training. Could the authors elaborate on the practical scenarios where this immediate advantage is particularly important?

  3. PERSIA minimizes the reconstruction error of client updates, but this objective does not necessarily align with personalized task performance. How do the authors interpret this relationship, especially given the degradation observed under heterogeneous local training budgets?

  4. The main personalized experiments focus on GLUE classification with RoBERTa-Large, while the generative-model experiments use a different global FL protocol. Could the authors clarify to what extent the observed aggregation benefits are expected to generalize to larger generative models under personalized FL?

5 comments
AuthorsOpenReview ↗

Response to Reviewer N8zs (1/3)

We thank the reviewer for the positive assessment and for recognizing PERSIA's principled subspace-based approach, its clear connection between the identified problem, the proposed solution, and its theoretical justification, and the reconstruction optimality guarantee, which strengthens the methodological contribution. We answer every question below. In particular, we added:

  • final accuracy against local training with six GLUE tasks and with 100 clients (W1 and Q1);
  • served accuracy, i.e., the accuracy of the models clients use between synchronizations, and runs with one or two aggregations (W2 and Q2);
  • reconstruction error versus accuracy, and a heterogeneous-budget rerun (Q3);
  • a personalized generative benchmark with Qwen2.5-3B-Instruct, including long local phases (Q4).

All changes are marked in blue in the revised paper, and their locations are given below.


W1 and Q1: Knowledge sharing beyond preventing aggregation loss.

Response to W1 and Q1:

When clients differ or are numerous, PERSIA outperforms local training (+1.79 with six tasks, +1.74 and +2.48 with 100 clients). For single-task clients with enough local data, it matches local training while preventing aggregation loss; with 100 clients, sharing both factors (FedAvg-LoRA (personal head)) helps more.

The submitted paper reported final accuracy on par with and above local training (Table 1, τ = 1280: PERSIA 91.93, FedAvg-LoRA 91.97, FedSA-LoRA 91.92, local 91.42) and, with six tasks, PERSIA ahead of FedSA-LoRA (85.79 ± 0.50 vs 84.93 ± 0.16).

(a) Single-task clients gain little from any one-sided rule (W1). With 10 clients on one task, each client's own data suffices: in the tuned rerun of Table 1, PERSIA ties local training (+0.23 [−0.32, +0.78] at τ = 1280; +0.23 [−0.42, +0.87] at τ = 40).


(b) Gain over local training (Q1) (final accuracy, 3 seeds):

SettingLocalPERSIAFedSA-LoRAFedAvg-LoRA (personal head)PERSIA − local
Six GLUE tasks, 12 clients (validation, pre-registered score, CoLA by MCC)79.49 ± 0.9681.27 ± 0.7280.28 ± 0.4979.29 ± 0.16 (retuned)+1.79 [+0.19, +3.38]
100 clients, 10 per round, MNLI84.35 ± 0.3186.10 ± 0.3286.42 ± 0.0890.76 ± 0.06+1.74 [+1.27, +2.22]
100 clients, QNLI87.10 ± 0.2489.59 ± 0.1789.35 ± 0.2693.42 ± 0.38+2.48 [+1.45, +3.52]

(c) What the gain depends on (Q1). With six tasks, PERSIA is ahead of FedSA-LoRA (+0.99 [+0.41, +1.57]) and ties FedAvg-LoRA (personal head) (+1.98 [−0.17, +4.14]). Served accuracy (W2) adds a second view: it measures how each client performs with the aggregated model it receives, without further training, i.e., the quality the system delivers to its users until the next synchronization. On served accuracy, PERSIA is above local training, FedSA-LoRA, and FedAvg-LoRA (personal head) (+1.08 [+0.24, +1.92], +1.37 [+1.01, +1.73], +3.10 [+1.75, +4.46]). With 100 clients, FedSA-LoRA gains as much as PERSIA, and FedAvg-LoRA (personal head), which also shares B, is about 4 points ahead (−4.67 [−5.41, −3.92] MNLI, −3.83 [−5.21, −2.45] QNLI; served −5.13 [−5.79, −4.47]). (3 seeds)


In the revised paper, Appendix H.1 provides a rerun of Table 1 (Table 21), and the new Appendix H.2 provides the six-task and 100-client results (Tables 25 and 26).

AuthorsOpenReview ↗

Response to Reviewer N8zs (2/3)

W2 and Q2: When does the advantage right after aggregation matter?

Response to W2 and Q2:

It matters when clients use the model between synchronizations, when only a few aggregations are possible, or when communication rounds are scarce. With three aggregations, PERSIA's clients hold models +1.43 points above FedSA-LoRA and +5.27 above FedAvg-LoRA (personal head), and with one aggregation, +13.60 above FedAvg-LoRA (personal head). With frequent communication, FedAvg-LoRA (personal head) is ahead; PERSIA targets settings where clients synchronize rarely.

The submitted paper reported accuracy right after the first aggregation (Table 1: PERSIA 91.09, local 90.82, FedSA-LoRA 86.68, FedAvg-LoRA 79.53) and named these scenarios (Section 6); the revision keeps these values and adds a rerun with Algorithm 1 as written and every method tuned (Table 21).

(a) Served accuracy (Q2) measures this scenario: each client's test accuracy at every logged (post-aggregation) evaluation, averaged over the run, then over clients; defined before any value was computed.


(b) Three aggregations of 1,280 local steps, four tasks (W2) (3 seeds):

Methodright after 1st aggregationservedfinal
Local91.16 ± 0.1491.38 ± 0.0491.73 ± 0.20
PERSIA90.87 ± 0.1491.48 ± 0.0591.96 ± 0.03
FedSA-LoRA86.98 ± 0.3490.05 ± 0.0891.92 ± 0.30
FedAvg-LoRA (personal head)76.60 ± 0.9486.20 ± 0.2691.95 ± 0.17

Served: PERSIA − FedSA-LoRA +1.43 [+1.18, +1.67], PERSIA − FedAvg-LoRA (personal head) +5.27 [+4.58, +5.96]; final accuracy ties (+0.04 [−0.75, +0.84], +0.02 [−0.47, +0.50]).


(c) Where it does not matter (Q2). With 80 aggregations of 40 steps, clients are realigned after every short phase, and FedAvg-LoRA (personal head) is ahead (served −1.37 [−1.45, −1.30], final −1.72 [−2.08, −1.35]). With frequent communication, we therefore recommend sharing both factors; with few synchronizations, PERSIA is the better choice. (3 seeds)


(d) With one or two aggregations, the lead grows (Q2). 3,200 local steps, MNLI/QNLI, learning rates retuned; served accuracy, PERSIA − X (3 seeds):

Aggregationsvs FedSA-LoRAvs FedAvg-LoRA (personal head)
2+2.54 [+0.95, +4.13]+11.02 [+9.26, +12.78]
1 (served = final)+2.89 [−1.67, +7.45]; added seeds +2.97 [+1.35, +4.58]+13.60 [+9.09, +18.11]

(Added seeds: PERSIA and FedSA-LoRA; 3 seeds primary.) The one-aggregation grid was extended below its smallest value after the selections sat at that edge (FedAvg-LoRA (personal head) moved on both tasks, FedSA-LoRA on MNLI; PERSIA's MNLI cell, also extended, kept its rate; before: +13.56, +2.60, added seeds +2.62). Our response to Reviewer qd2D (W3 and Q3) gives the full table.

In the revised paper, Appendix H.1 gives these results (Tables 22–24).

AuthorsOpenReview ↗

Response to Reviewer N8zs (3/3)

Q3: Reconstruction error versus personalized accuracy, and heterogeneous budgets.

Response to Q3:

PERSIA's objective predicts the accuracy degradation at aggregation (median Spearman 0.86), and the drop under heterogeneous budgets that the reviewer notes does not come from aggregation; it comes from the limited rebase residual from clients with the smallest budgets. The results in Table 11 from the originally submitted paper report PERSIA 88.29 vs FedSA-LoRA 91.33 on QNLI under severe budgets with one learning rate shared by every method; a more complete, tuned rerun presented in the revision finds PERSIA on par with FedSA-LoRA.

(a) Error versus accuracy (Q3). We applied seven aggregation methods (PERSIA, PERSIA without whitening, FedSA-LoRA, factor averaging with rebasing, projector consensus, FedRot-LoRA, and two-factor averaging) to the same client models at the first aggregation of 24 training runs: methods with lower reconstruction error lose less accuracy (median Spearman correlation 0.86; Appendix H.3). Local training repairs these losses (W2).


(b) Heterogeneous budgets (Q3). With one learning rate shared by every method and the validation metric fixed in advance, PERSIA is on par with FedSA-LoRA (−0.06 [−0.58, +0.46] severe, −0.01 [−0.91, +0.89] extreme). Against a uniform budget, the 5-step clients drop by 3.81 [+0.15, +7.47] under PERSIA and by 3.16 [−0.38, +6.69] under FedSA-LoRA; these clients have the smallest rebase residual, so their loss does not stem from how the aggregation reconstructs their updates. Our response to Reviewer qd2D (W4 and Q4) provides results for all budget groups. (3 seeds)

In the revised paper, the new Appendix H.5 gives these results (Tables 34 and 35).


Q4: Generalization to larger generative models under personalized FL.

Response to Q4:

The aggregation carries over unchanged, and on a personalized LLM benchmark, it preserves what clients learn. Proposition 3.1 holds for any adapted matrix. On Qwen2.5-3B-Instruct, right after an aggregation, PERSIA keeps the clients' own accuracy (−0.22 [−1.36, +0.92] against their local models) and is +7.14 [+5.89, +8.39] above two-factor averaging; over the run, its served accuracy is +3.19 [+2.31, +4.06] above FedAvg-LoRA. A study with larger models and larger client datasets did not fit within the rebuttal period, particularly given the large number of clients with diverse datasets that match the PERSIA use case.

The submitted paper evaluated a generative model on the global protocol (GSM8K, 128 held-out problems); the revision adds the full 1,319-problem test set (Table 40).

(a) Personalized benchmark (Q4). Qwen2.5-3B-Instruct, eight clients with different FedDPA tasks, every method tuned over 10 configurations; three rounds of 68 steps. Final accuracy: PERSIA 76.58 ± 1.15, FedSA-LoRA 77.14 ± 0.65, local 76.58 ± 0.65, FedAvg-LoRA 73.64 ± 0.63. Served accuracy: PERSIA is +3.19 [+2.31, +4.06] above FedAvg-LoRA and level with FedSA-LoRA (−0.38 [−0.92, +0.16]) and local training (−0.52 [−1.14, +0.11]). With 40 rounds of 25 steps, FedSA-LoRA is ahead (80.00 versus 68.17 ± 8.01, Table 36).


(b) Long local phases (Q4). One 1,280-step phase, every rule applied to the same client models. PERSIA ties the clients' local models (−0.22 [−1.36, +0.92]) and leads two-factor averaging by +7.14 [+5.89, +8.39] (+5.69 [+4.11, +7.28] after 340 steps); against FedSA-LoRA it is level (+0.22 [−0.95, +1.40]; with FedSA-LoRA's own hyperparameters +0.28 [−1.28, +1.83]). Each client has only 270 training items, so 1,280 steps are about 19 passes over its data; the GLUE gap (+3.89) arises with far larger clients and is not reproduced here. With FedAvg-LoRA’s hyperparameters, local training itself collapses below zero-shot (49.42 vs 68.67); there, averaging partly repairs the collapsed models, while PERSIA preserves each client’s own model (Appendix H.6).

In the revised paper, the new Appendix H.6 gives both experiments (Tables 36–38) and the GSM8K test set (Table 40).


We again thank the reviewer for the positive assessment. The revision now adds results for the six-task, 100-client, served-accuracy, heterogeneous-budget, and personalized LLM settings.

The reviewer's question about when post-aggregation accuracy matters prompted a direct evaluation under sparse synchronization. With three aggregations, PERSIA improves served accuracy by +1.43 points over FedSA-LoRA and +5.27 points over FedAvg-LoRA with a personal head; with six heterogeneous tasks, its final accuracy is also above local training (+1.79).

If these results resolve the reviewer's concerns, we would be grateful if the reviewer would reconsider their evaluation and support the paper in the discussion. We welcome any remaining questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful and constructive reviews. They describe PERSIA as a principled subspace-based approach with a reconstruction optimality guarantee (N8zs), addressing a problem that is clearly defined with low communication overhead (qd2D), as intuitively clever and mathematically elegant (EpMB), as addressing a more fundamental issue than standard parameter-space client drift (ELcy), and as having significantly mitigated the decline in post-aggregation instantaneous accuracy (GXNd).

The new results tell one consistent story. Gram whitening makes PERSIA's aggregation exactly optimal and independent of how clients factorize their updates: on identical client models it loses 0.05 MNLI points at an aggregation, against 1.07 without whitening and 3.00–8.61 for the other one-sided rules. When communication is sparse, this carries into the models clients use; with six heterogeneous tasks (τ = 40), PERSIA's final accuracy is above both local training (+1.79) and FedSA-LoRA (+0.99).

We revised the paper and marked all changes in blue. We kept every printed result; the new results are in the new Appendix H, which a new Section 6 paragraph summarizes.

Requested byWhat was addedMain result
N8zs, qd2D, ELcy, GXNdServed accuracy (the model each client holds, averaged over the run); 1–3 aggregations; equal uplink (H.1)Three aggregations: served +1.43 vs FedSA-LoRA and +5.27 vs FedAvg-LoRA (personal head), at comparable final accuracy. One aggregation: +13.60 vs FedAvg-LoRA (personal head), and vs FedSA-LoRA +2.97 with two added seeds (+2.89 [−1.67, +7.45] with three). Equal uplink: +13.65
N8zs, ELcy, GXNdSix tasks, 100 clients, natural domain shift (H.2)+1.79 over local (six tasks); +1.74 / +2.48 over local with 100 clients; genres: served +1.54 vs FedSA-LoRA and +16.53 vs FedAvg-LoRA (personal head), at comparable final accuracy
qd2D, GXNdClosest one-sided rules on identical client models (H.3)MNLI points lost at one aggregation: PERSIA 0.05, projector consensus 5.69, FedRot-LoRA 8.53
qd2D, ELcy, GXNdFedALT, PF2LoRA, PERSIA inside FedALT (FALT-P); costs (H.4)FedALT: higher mean in all four cells; ahead at both schedules with two added seeds (+1.33, +1.14; three seeds +1.13, +1.37), at half the uplink. PF2LoRA: on par with PERSIA under sparse communication (τ = 1280), ahead with frequent synchronization (τ = 40), at 2× PERSIA's uplink and with second-order computation. FALT-P: on par with FedALT without retuning
N8zs, qd2D, GXNdCompute-heterogeneity rerun, reweighting, per-budget results (H.5)With one learning rate shared by every method, PERSIA and FedSA-LoRA are on par; accuracy lost at an aggregation follows reconstruction error (median Spearman 0.86)
N8zs, ELcyGenerative LLM (Qwen2.5-3B-Instruct), personalized protocol (H.6)Preserves what clients learn (+7.14 over two-factor averaging after a long phase; served +3.19 over FedAvg-LoRA); on par with FedSA-LoRA after three rounds and after a long phase
EpMBWhat Zᵢ reveals, secure aggregation, DP (H.7); streamed server step (H.8); rank 64 and 128 (H.9)Compatible with secure aggregation and record-level DP-SGD at FedSA-LoRA's privacy cost; error below FedSA-LoRA's under noise up to σ = 0.5; exact streamed server step, 17.5 ms per module at N = 10,000; at r = 64 PERSIA stays at local accuracy right after aggregation, while FedAvg-LoRA (personal head) loses 15.10 points more
qd2D, ELcy, GXNdEckart–Young–Mirsky basis, identifiability, why only Vᵣ, clients in different subspaces, and scope and extensions of Theorem 4.4 (H.10)Exact reduction and identifiability; least total loss at any subspace overlap; Theorem 4.4 extended to partial participation and inexact subspaces

Protocol: RoBERTa-Large, GLUE, 10 clients unless stated; PERSIA is Algorithm 1 as written; 3 seeds, mean ± sample std, seed-paired 95% t-intervals (a lead is claimed only when the whole interval has one sign); seeds 3–4 only where marked, always next to the 3-seed result. At one aggregation, the learning-rate grid was extended below its smallest value after the selections sat at that edge (FedAvg-LoRA (personal head) moved on both tasks, FedSA-LoRA on MNLI; PERSIA's MNLI cell kept its rate); before: +13.56 and +2.60 (+2.62 with added seeds); equal uplink +13.61.


The remaining edits (abstract and Conclusion unchanged):

  • Section 6, Appendix C.4, Table 11, and Remark F.6: with Algorithm 1 and one learning rate for every method, PERSIA matches FedSA-LoRA under heterogeneous budgets, and reweighting is not needed.
  • Section 3: the randomized SVD is stated with oversampling, and an exact streamed server step is added.
  • Table 1: Table 21 adds a rerun with every method tuned, including FFA-LoRA, FedEx-LoRA, and FedProx with Bᵢ local.
  • Appendix B.3: ε = 1e-12 and the server-side error-feedback buffer; Appendix H.6: the full 1,319-problem GSM8K test set.

Dear Reviewers,

Thank you for your time and feedback. As the discussion period draws to a close, please take a moment to reply to the authors if you have yet to comment on their response or follow up on their latest reply.

Please let them know which concerns have been addressed and which, if any, remain. If you have no further questions, a brief confirmation is sufficient.

Best,
Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.