- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PERSIA, a consistent subspace aggregation method for personalized federated LoRA, where clients share the right factors while keeping their left factors local. PERSIA uses Gram-based transformations and truncated SVD to construct a shared right subspace, followed by client-side rebasing to preserve personalized updates. The authors establish an optimality guarantee for reconstruction error and provide a factor-stationarity analysis.
Critical strengths
Over longer local training intervals, PERSIA significantly mitigated the decline in post-aggregation instantaneous accuracy, validating the effectiveness of the proposed mechanism in improving aggregation consistency.
Critical weaknesses
1.The method’s motivation and practical necessity need to be further strengthened. This paper is based on the one-sided parameter-sharing setup of FedSA-LoRA; however, given that the communication overhead of the B matrix in LoRA is relatively limited, the practical benefits of omitting the transmission of B still require more thorough justification. Furthermore, the design of this method relies on SVD reconstruction on the server side, but the validity of this objective decomposition still needs further explanation.
2.The overall performance gains and robustness are limited. Although PERSIA demonstrates outstanding performance in terms of real-time accuracy after aggregation, its overall performance did not outperform existing baselines in multiple experiments. The method’s performance deteriorates under more heterogeneous client conditions. Experimental comparisons with representative recent federated fine-tuning methods remain limited.
Questions for authors
- The authors need to further explain the rationale for decomposition on the server side and why they chose to retain the V-matrix.
- The authors need to provide additional explanation regarding the robustness of the experimental results.
6 comments
Response to Reviewer GXNd (2/3)
W2: Overall gains and robustness are limited; PERSIA deteriorates under more heterogeneous clients; few comparisons with recent methods.
Response to W2:
PERSIA's gain is in the model clients use: with sparse synchronization, its served accuracy is +1.43 to +16.53 points above FedSA-LoRA and FedAvg-LoRA (personal head), also under natural domain shift, while final accuracy is on par with FedSA-LoRA. Under compute heterogeneity, it also matches FedSA-LoRA. Among recent methods, PERSIA's mean is above FedALT's at both schedules (clearly so with 5 seeds) and, with sparse communication, matches PF2LoRA at half the uplink.
The submitted paper reported final accuracy on par with the baselines (Table 1: at τ = 1280, PERSIA 91.93, FedAvg-LoRA 91.97, FedSA-LoRA 91.92; at τ = 40, 91.75, 92.65, and 91.80), PERSIA within 1.2 points of FedSA-LoRA across label skew, client count, rank, and participation (Table 3), and a QNLI deficit under compute heterogeneity (Table 11: 88.29 vs 91.33).
(a) Final accuracy (W2). In the tuned rerun of Table 1, PERSIA is on par at τ = 1280 (vs FedSA-LoRA +0.04 [−0.75, +0.84]); with frequent synchronization (τ = 40) two-factor sharing is ahead (−1.72 [−2.08, −1.35]). (3 seeds)
(b) Served accuracy (W2) (the model each client holds, averaged over the run; defined in advance), PERSIA − X (3 seeds):
| Setting | vs FedSA-LoRA | vs FedAvg-LoRA (personal head) |
|---|---|---|
| 3 aggregations of 1,280 steps, 4 tasks | +1.43 [+1.18, +1.67] | +5.27 [+4.58, +5.96] |
| 2 aggregations of 1,600 steps, retuned | +2.54 [+0.95, +4.13] | +11.02 [+9.26, +12.78] |
| 1 aggregation of 3,200 steps, retuned (= final) | two added seeds +2.97 [+1.35, +4.58]; three seeds +2.89 [−1.67, +7.45] | +13.60 [+9.09, +18.11] |
(c) Heterogeneous tasks and natural domain shift (W2). Six GLUE tasks: final +1.79 [+0.19, +3.38] over local and +0.99 [+0.41, +1.57] over FedSA-LoRA, on par with retuned FedAvg-LoRA (personal head) (+1.98 [−0.17, +4.14]); served, ahead of all three. MultiNLI genres (no label skew, learning rates not retuned), three aggregations: served +1.54 [+0.41, +2.66] over FedSA-LoRA and +16.53 [+13.83, +19.24] over FedAvg-LoRA (personal head), at comparable final accuracy (+0.39 [−0.73, +1.51]; +1.59 [−8.45, +11.64]). Our responses to Reviewers N8zs (W1 and Q1) and ELcy (W2) give the tables. (3 seeds)
(d) Compute heterogeneity (W2). In a rerun of Table 11 with one learning rate shared by every method, PERSIA and FedSA-LoRA are on par on the pre-registered validation metric (−0.06 [−0.58, +0.46] severe, −0.01 [−0.91, +0.89] extreme; test +0.69 and −0.90). Our response to Reviewer qd2D (W4 and Q4) gives the reweightings and per-budget results. (3 seeds)
(e) Recent methods (W2). FedALT [1]: with two added seeds (PERSIA and FedALT only), PERSIA − FedALT is +1.33 [+0.03, +2.64] (τ = 40) and +1.14 [+0.24, +2.04] (τ = 1280); with three seeds +1.13 [−1.92, +4.18] and +1.37 [−0.86, +3.61], a higher mean in all four task–schedule cells. PF2LoRA [2] (our implementation): with sparse communication (τ = 1280) the two are on par (91.24 ± 0.03 vs 90.94 ± 0.48; +0.29 [−0.98, +1.56]); with frequent synchronization (τ = 40) PF2LoRA is ahead (92.57 ± 0.42 vs 90.89 ± 0.32; −1.67 [−2.50, −0.85]), at the cost of a second adapter with second-order updates (2.1–2.8× the GPU time of FedAvg-LoRA (personal head), shared GPUs, upper bounds). FedALT took 1.4–2.3× the wall-clock time. Both upload twice PERSIA's values. Closest one-sided rules, on identical client models at the first τ = 1280 aggregation: PERSIA loses 0.05 MNLI points, projector consensus 5.69, FedRot-LoRA 8.53. Our response to Reviewer qd2D (W1 and Q1; W2 and Q2) gives the tables and costs. (3 seeds)
In the revised paper, Appendix H.1 gives the rerun of Table 1 and served accuracy (Tables 21–24), Appendix H.2 the heterogeneous-task and domain-shift results (Tables 25 and 27), Appendix H.3 the closest rules (Table 28), Appendix H.4 FedALT and PF2LoRA (Tables 30 and 33), and Appendix H.5 the rerun (Tables 34 and 35); Section 6 points to each.
[1] Bian, Jieming, et al. "Fedalt: Federated fine-tuning through adaptive local training with rest-of-world lora." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 24. 2026.
[2] Hao, Jie, et al. "Personalized federated fine-tuning for heterogeneous data: An automatic rank learning approach via two-level lora." arXiv preprint arXiv:2503.03920 (2025).
Response to Reviewer GXNd (3/3)
Q2: Additional explanation regarding the robustness of the experimental results.
Response to Q2:
Every new comparison reports three seeds and a stated tuning rule, with seed-paired 95% intervals wherever per-seed values exist; the main comparisons tune every method by one procedure with selection on validation and served accuracy, and the domain-shift and LLM analyses were defined before any value was computed. The current code reproduces the submission-era runs we re-ran record for record.
The submitted paper reported the mean ± population standard deviation over three seeds and the per-client majority floor, excluded RTE from the averages (Section 5), and described the rebase guard for collapsed local phases (Appendix B.3).
(a) Statistics (Q2). Final-round unweighted client means, mean ± sample std over three seeds (for PERSIA vs FedALT, and vs FedSA-LoRA with one aggregation, also two added seeds, reported next to the 3-seed result), and seed-paired 95% t-intervals; a lead is claimed only when the whole interval has one sign. We defined and logged served accuracy before computing any value.
(b) Tuning (Q2). The rerun of Table 1 and the fixed-budget, FedALT, rank-sweep, and LLM comparisons tune each compared method, PERSIA included, by one procedure (same search, selection on validation only; test never used for selection); the adapted FedRot-LoRA and projector consensus, and PF2LoRA, got a smaller search. The six-task, data-scarce, 100-client, and compute-heterogeneity runs use one learning rate (2e-4) for every method, with FedAvg-LoRA (personal head) retuned in the six-task and data-scarce settings, where PERSIA led; r = 64 uses the learning rate selected at r = 32.
(c) Training stability (Q2). In the 12 tuned MNLI/QNLI runs of PERSIA, no client ends at its majority class; in the tuned rank sweep, PERSIA and FedSA-LoRA each leave 2 clients there over 18 runs. At a fixed, untuned learning rate a client can stall there under one-sided rules; the rebase guard of Appendix B.3 is designed for this case.
(d) Reproduction (Q2). The current code reproduces the submission-era runs we re-ran record for record (120 of 120 server evaluations of global-protocol PERSIA without error feedback).
In the revised paper, the opening paragraphs of the new Appendix H (protocol, hyperparameter selection, and reliability) give these details.
We thank the reviewer for prompting a clearer explanation of what the one-sided design offers beyond bandwidth savings. Under sparse synchronization, PERSIA exceeds two-factor sharing by 14.26 points immediately after aggregation and, with data-scarce clients, by 5.78 (MNLI) and 2.07 (QNLI) points in final accuracy.
The revision also explains why the server objective is determined by the client messages and why only the shared right subspace is broadcast. It adds served-accuracy, heterogeneity, compute-heterogeneity, FedALT, and PF2LoRA evaluations, together with the tuning procedures and statistical support for the new results.
If these results and clarifications resolve the reviewer's concerns, we would be grateful if the reviewer would reconsider their evaluation. We would be happy to clarify any remaining points.
Response to Reviewer GXNd (1/3)
We thank the reviewer for the careful assessment and for recognizing that PERSIA significantly mitigated the decline in post-aggregation instantaneous accuracy, validating the effectiveness of the proposed mechanism in improving aggregation consistency, and that it demonstrates outstanding performance in terms of real-time accuracy after aggregation. We answer every question below. In particular, we added:
All changes are marked in blue in the revised paper, and their locations are given below.
Response to W1 (motivation):
The benefit is personalization, not bandwidth: no aggregation overwrites a client's Bᵢ, and when communication is sparse, this keeps the clients' models accurate. Right after aggregation, PERSIA's clients are +14.26 points above two-factor sharing; with six tasks, two-factor sharing does not improve on local training (79.29 vs 79.49), while PERSIA does (+1.79).
The submitted paper adopted FedSA-LoRA's one-sided design (A captures common knowledge, B adapts to local data; Section 1) and reported 0.76 MiB per round for PERSIA against 1.50 MiB for two-factor sharing (Table 12).
(a) Personalization (W1). As in FedSA-LoRA, no aggregation overwrites a client's Bᵢ, and the server never receives Bᵢ or BᵢAᵢ; omitting Bᵢ also saves 0.75 MiB per client and round.
(b) What this buys (W1), against FedAvg-LoRA (personal head), which shares both factors (3 seeds):
The General response describes the one-aggregation learning-rate grid.
In the revised paper, Appendix H.7 details what the server learns from the upload, and Appendices H.1 and H.2 give these comparisons.
Response to W1 (objective) and Q1:
The objective is valid: it is the personalized aggregation error, which these client messages determine, and it tracks the accuracy lost at aggregation (median Spearman 0.86). The server computes Vᵣ because it depends on all clients, and Vᵣ is the only quantity all clients share.
The submitted paper defined the objective (eq. 2), made it computable from Zᵢ (eqs. 3–4), and proved that the top-r right singular vectors Vᵣ minimize it (Proposition 3.1).
(a) What it optimizes (W1). Since ZᵢᵀZᵢ = ΔWᵢᵀΔWᵢ, minimizing Σᵢ‖ΔWᵢ − Bᵢ′A′‖² over a shared A′ and private Bᵢ′ is a principal-subspace problem on the stack Z̃; the bound is Eckart–Young–Mirsky on Z̃, and each client attains it by rebasing. It is also the aggregation term of Theorem 4.4, which PERSIA minimizes every round.
(b) Why this objective (W1). (RᵢBᵢ, Aᵢ) with any orthogonal Rᵢ sends the same message, so no rule whose messages are functions of Aᵢ and BᵢᵀBᵢ (as PERSIA's and FedSA-LoRA's are) can recover the dense mean or its subspace. These messages do determine Σᵢ ΔWᵢᵀΔWᵢ, hence our objective; factor averaging instead depends on each client's arbitrary coordinates.
(c) It tracks the accuracy lost at an aggregation (W1). Across seven rules on identical client models, the accuracy lost follows the objective (median Spearman 0.86 over 22 runs; pre-specified), and lower error shows in served accuracy (W2).
(d) Why the server, and why V (Q1). The shared subspace depends on all clients, so the server computes it; rebasing needs Bᵢ, so the client does it. Write Z̃ = UΣVᵀ. Block i of UᵣΣᵣ = Z̃Vᵣ is ZᵢVᵣ = OᵢᵀBᵢ⋆, with Oᵢ the polar factor of Bᵢ: client i's rebased factor in a private frame, which the client computes itself as Bᵢ⋆ = ΔWᵢVᵣ. So U carries nothing shared, Σ is absorbed into Bᵢ⋆, and Vᵣ is the only shared quantity on the input space. Any A′ = MVᵣᵀ with Bᵢ′ = Bᵢ⋆M⁻¹ gives the same products; the orthonormal choice makes the rebase a plain product.
In the revised paper, Appendix H.10 states the Eckart–Young–Mirsky basis, identifiability (Remark H.1), and why only Vᵣ is needed, and Appendix H.5 gives the error-versus-accuracy analysis.