- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates the separation of shared and personalized knowledge in federated LoRA under conditions of data heterogeneity. The authors propose GReP-LoRA, which initializes shared LoRA factors using globally averaged gradients and constructs a personalized adaptation space based on client-side gradient residuals. Additionally, the method employs a first-order approximation of shared factor updates to facilitate federated aggregation and validates its effectiveness on the DomainNet, NICO++, and GLUE datasets.
Critical strengths
The authors point out that a high degree of similarity among the full LoRA factors does not necessarily imply that their effective update directions are consistent, and that this similarity may be influenced by the initialization method. This finding offers a valuable perspective for rethinking parameter-sharing strategies in federated LoRA.
Critical weaknesses
1.The selection of private spaces lacks sufficient validation of necessity. One of the core design elements of this paper is the use of client gradient residuals to select private adaptation spaces. The advantages of this mechanism require further exploration. Furthermore, since the private spectral basis remains frozen during subsequent training, it is difficult to ensure that the selected spaces can continuously adapt to the ever-changing directions of client gradients during the training process.
2.There is an optimization inconsistency between local linearization training and server-side model reconstruction. This paper uses a first-order approximation to replace the standard bilinear LoRA update, omitting the local interaction terms (). However, server-side aggregation reconstruction introduces new second-order interaction terms. This may lead to discrepancies between the locally optimized model and the final global model.
Questions for authors
1.Compared to directly using local gradients or other gradient initialization strategies, what unique advantages does the use of gradient residuals to select a private adaptation space offer in terms of performance on real-world tasks?
2.Since the private spectral basis is determined only during the initialization phase, how sensitive is this method to changes in the direction of client gradients during subsequent training?
3.How significant are the second-order interaction terms introduced by server-side factor reconstruction in actual training? How do they affect model convergence and final performance?
7 comments
Response for Reviewer 9W1N (2/5)
B. Rationale and evidence for residual basis selection
Residual-based basis selection targets the client correction after the common update. At a common reference model, let the client gradient be , the weighted average gradient be , and the residual be . Taking one shared full-gradient step as the reference, a private correction of rank at most has the following approximation objective:
\left\\|-\eta\bar G+D_i-(-\eta G_i)\right\\|_F^2 =\min_{\operatorname{rank}(D_i)\le r_p}\\|D_i+\eta R_i\\|_F^2.$$ By the Eckart–Young–Mirsky theorem, exact truncated SVD gives $$D_i^\star=-\eta[R_i]_{r_p}, \qquad \min_{\operatorname{rank}(D_i)\le r_p}\\|D_i+\eta R_i\\|_F^2 =\eta^2\sum_{j > r_p}\sigma_j(R_i)^2.$$ Thus, **the leading residual spectral directions correspond to the optimal client correction under this reference and rank budget**. Raw local gradients contain both common components and client differences; residual-based basis selection concentrates private capacity on corrections relative to the common direction. This result also allows comparison of different spaces with the same dimensions. For any left and right bases with $r_p$ orthogonal columns, denoted by $P,Q$, the optimal residual representation error is $$\mathcal E(P,Q) =\min_C\\|R_i-PCQ^\top\\|_F^2 =\\|R_i\\|_F^2-\\|P^\top R_iQ\\|_F^2.$$ If $P_R,Q_R$ are the leading residual spectral bases, then $$\mathcal E(P,Q)-\mathcal E(P_R,Q_R) =\sum_{j\le r_p}\sigma_j(R_i)^2-\\|P^\top R_iQ\\|_F^2 \ge0.$$ This optimality applies to residual representation at a common reference point. At the same capacity and shared initialization (Table 1(a)), residual bases improve average accuracy over raw local gradient bases by **0.3044**, perform better in five of six domains, and reduce CE from **0.3480** to **0.3420**. To test the benefit at lower private capacity, Table 2(a) compares residual and raw local gradient bases at private rank 4. **Table 2(a). Basis selection comparison with lower private capacity. NICO++, 30 clients, shared rank 16/private rank 4, round 30, accuracy (%).** | Basis source | Autumn | Dim | Grass | Outdoor | Rock | Water | Average | CE ↓ | |---|---:|---:|---:|---:|---:|---:|---:|---:| | Residual gradients | **93.35** | **89.31** | 94.61 | **91.61** | **91.10** | **90.42** | **91.73** | **0.3628** | | Raw local gradients | 93.13 | 89.19 | **94.66** | 91.43 | 90.60 | 90.24 | 91.54 | 0.3671 | At private rank 4, residual bases improve average accuracy by **0.1916**, reduce CE by **0.004330**, and perform better in five of six domains, including a **0.4971** gain on Rock. The benefit therefore persists at lower capacity. To test whether these directions capture client differences across samples, we select bases using 1,024 training images per client and measure residual coverage on 1,024 disjoint training images in DomainNet with 6 clients. All comparisons use the same fixed reference model and private rank 16. **Table 2(b). Residual energy coverage on independent samples. DomainNet, 6 clients, private rank 16.** | Basis source | Residual energy coverage ↑ | |---|---:| | Own residual gradients | **57.21%** | | Raw local gradients | 43.49% | | Population average gradient | 40.47% | | Random orthogonal bases | 0.02990% | Residual bases improve coverage by **13.72** over raw local gradient bases. Together with the task results at equal capacity, this supports their ability to capture reproducible client corrections. This allocation of private capacity complements shared initialization methods such as LoRA-GA and OLoRA, which set the common adapter's starting point.Response for Reviewer 9W1N (3/5)
C. Continued adaptation within fixed spaces
The trainable core continually adapts to current gradients, and fixed bases retain predictive value in later training. Absorbing the fixed scaling into the basis matrices, the private weights and their core gradient are
\qquad \nabla_{U_i}f_i=P_i^\top(\nabla_{E_i}f_i)Q_i.$$ With both bases fixed, each update of $U$ uses the current task gradient to recombine directions within the space. Holding the other parameters fixed, let $h_i(U)=f_i(W_{\mathrm{shared}}+P_iUQ_i^\top)$. If $h_i$ is $L_U$-smooth and a full-gradient step $U^+=U-\eta\nabla h_i(U)$ is taken with $0 < \eta\le1/L_U$, then $$h_i(U^+)\le h_i(U)-\frac{\eta}{2}\\|\nabla h_i(U)\\|_F^2.$$ Under these conditions, updating $U$ decreases the objective when the current gradient has a nonzero component in the private space. To test the value of combining directions, Table 3(a) compares independently trained full and diagonal cores. **Table 3(a). Independently trained full versus diagonal cores. NICO++, 30 clients, shared/private rank 16, round 15, accuracy (%).** |Privatecore|Autumn|Dim|Grass|Outdoor|Rock|Water|Average|CE↓| |-|-:|-:|-:|-:|-:|-:|-:|-:| |FullU|**93.73**|**90.71**|**94.17**|91.52|**91.78**|**90.58**|**92.08**|**0.3430**| |DiagonalU|93.46|89.63|93.61|**91.91**|91.19|90.42|91.70|0.3464| The full core improves average accuracy by **0.3793**, performs better in five of six domains, and lowers CE. Both effective private matrices have maximum rank 16, with 6,400 versus 400 trainable core parameters. The comparison supports the benefit of a full core, which offers cross-direction combinations and greater capacity. **Gradient drift and refreshing.** To characterize subsequent gradient drift, let $\mathcal P_i$ be the orthogonal projection onto the initial private space. If the residual gradient is $L_{R,i}$-Lipschitz in the neighborhood under consideration, then $$\\|(I-\mathcal P_i)R_i(W_t)\\|_F \le\\|(I-\mathcal P_i)R_i(W_0)\\|_F +L_{R,i}\\|W_t-W_0\\|.$$ Here, $W$ contains the parameters affecting the gradient. The bound separates initial projection error from displacement-induced gradient drift. To assess the value of updating the private space, we compare fixed bases with periodic refreshes. All groups use the same shared initialization and 128 samples per client for initial basis selection. Each refresh uses the same sample budget and projects the existing private weights into the re-estimated residual space. **Table 3(b). Fixed initial bases versus periodic refreshes. NICO++, 30 clients, shared/private rank 16, round 30, accuracy (%).** |Refreshpolicy|Extrarefreshes|Autumn|Dim|Grass|Outdoor|Rock|Water|Average|CE↓| |-|-:|-:|-:|-:|-:|-:|-:|-:|-:| |Fixedinitialbases|**0**|93.95|**90.27**|94.34|**91.91**|90.96|**90.86**|**92.05**|0.3763| |Refreshevery10rounds|2|93.90|90.03|**94.69**|91.61|**91.28**|90.71|92.04|0.3711| |Refreshevery5rounds|5|**94.39**|89.67|94.64|91.46|**91.28**|90.65|92.02|**0.3703**| Fixed bases achieve **0.01510** and **0.03450** higher average accuracy than refreshes every 10 and 5 rounds, respectively. Refreshes improve some domains and CE. Fixed bases retain the highest endpoint average accuracy without repeated gradient estimation, spectral decomposition, or core transfer. **Predictive contribution later in training.** To test whether fixed private spaces remain useful later in training, we compare inference with and without the private branch at the same NICO++ round-30 and DomainNet round-50 checkpoints. **Table 3(c). Private branch intervention on the fixed-basis model at round 30. NICO++, 30 clients, accuracy (%).** |Inferencesetting|Autumn|Dim|Grass|Outdoor|Rock|Water|Average|CE↓| |-|-:|-:|-:|-:|-:|-:|-:|-:| |Ownprivatebranch|**93.95**|**90.27**|**94.34**|**91.91**|**90.96**|**90.86**|**92.05**|**0.3763**| |Privatebranchoff|89.72|85.91|92.79|89.45|89.43|87.63|89.16|0.4934| At NICO++ round 30, the private branch improves average accuracy by **2.895**, with gains for all 30 clients. The initial space and trained $U$ therefore retain predictive value. **Table3(d).Predictivecontributionofthefixedprivatebranchatround50.DomainNet,30clients,accuracy(%).** |Inferencesetting|Clipart|Infograph|Painting|Quickdraw|Real|Sketch|Average|CE↓| |-|-:|-:|-:|-:|-:|-:|-:|-:| |Ownprivatebranch|**83.84**|**53.51**|**80.09**|**75.63**|**90.39**|**77.74**|**76.87**|**1.275**| |Privatebranchoff|80.65|41.80|74.73|67.11|89.39|71.59|70.88|1.882| At DomainNet round 50, the gain is **5.988**, with improvements in all six domains; Infograph, Quickdraw, and Sketch gain **11.71, 8.520, and 6.154**, respectively, and CE falls from **1.882** to **1.275**. The refresh and checkpoint experiments support **fixed residual bases with a trainable $U$ as a practical balance between adaptation and computational cost in these settings**.
Response for Reviewer 9W1N (1/5)
We thank you for the constructive suggestions on private space selection, continued adaptation, and consistency between local optimization and server reconstruction. We organize our responses into two groups: W1, Q1, and Q2 address the necessity of private spaces, residual-based basis selection, and subsequent adaptation; W2 and Q3 address server reconstruction and the practical impact of second-order terms. We explain the design rationale through structural ablations, basis selection comparisons with the same capacity, dynamic refresh experiments, checkpoint interventions, and full training experiments.
Thank you for suggesting further validation of the contribution of private spaces and their capacity for continued adaptation. We address the necessity of private adaptation, the rationale and evidence for residual basis selection, and continued adaptation within fixed spaces.
A. Necessity of private adaptation
Residual spectral bases allocate private capacity to client-specific directions, while the trainable core U adapts their combinations to current task gradients. To test private adaptation and basis selection, Table 1(a) compares the full method with independently trained alternatives.
Table 1(a). Ablation of private structures and basis selection. NICO++, 30 clients, shared rank 16/private rank 16, accuracy (%).
The full method improves average accuracy by 2.596 over no private branch, with gains in all six domains, and by 2.703 over random bases with the same U capacity.
To test whether adaptation is client-specific, we fix the shared A, B, and classification head at round 15 and either disable the private branch or swap complete branches between same-domain clients using a fixed mapping.
Table 1(b). Private branch interventions at the same round-15 checkpoint. NICO++, 30 clients, accuracy (%).
Disabling the branch reduces average accuracy by 3.441; swapping its frozen bases and U reduces accuracy by 6.177. Thus, the learned private branch contributes to prediction and is matched to individual clients, even within the same domain.