acceptodds
Under review as a conference paper at ICLR 2027

Similar Factors, Different Updates: Rethinking What to Share in Federated LoRA

Abstract

Low-rank adaptation (LoRA) enables efficient fine-tuning of foundation models in federated learning. However, determining what to share across clients remains challenging under data heterogeneity. Factor similarity provides an intuitive basis for selective sharing, but similar factors may reflect common initialization and accumulated server history rather than shared knowledge. We show that a single gradient step can produce perfectly aligned factors with opposite effective updates, revealing a fundamental limitation of similarity-based interpretations. Motivated by this finding, we propose GReP-LoRA, which uses task gradients to guide shared and personalized adaptation. Our method initializes a shared adapter from the population gradient and selects frozen private spectral bases from client residual gradients. Clients jointly learn shared tangent increments and compact private cores that rescale and combine the selected directions, while only shared parameters are aggregated. Our theoretical analysis connects captured residual energy to personalized objective improvement under quadratic assumptions and establishes nonconvex stationarity bounds with explicit optimization errors. We evaluate our method on vision and language tasks. Held-out gradient prediction and private-branch transfer experiments demonstrate client-specific specialization, while controlled comparisons support the usefulness of a full trainable core.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9W1NOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the separation of shared and personalized knowledge in federated LoRA under conditions of data heterogeneity. The authors propose GReP-LoRA, which initializes shared LoRA factors using globally averaged gradients and constructs a personalized adaptation space based on client-side gradient residuals. Additionally, the method employs a first-order approximation of shared factor updates to facilitate federated aggregation and validates its effectiveness on the DomainNet, NICO++, and GLUE datasets.

Critical strengths

The authors point out that a high degree of similarity among the full LoRA factors does not necessarily imply that their effective update directions are consistent, and that this similarity may be influenced by the initialization method. This finding offers a valuable perspective for rethinking parameter-sharing strategies in federated LoRA.

Critical weaknesses

1.The selection of private spaces lacks sufficient validation of necessity. One of the core design elements of this paper is the use of client gradient residuals to select private adaptation spaces. The advantages of this mechanism require further exploration. Furthermore, since the private spectral basis remains frozen during subsequent training, it is difficult to ensure that the selected spaces can continuously adapt to the ever-changing directions of client gradients during the training process.

2.There is an optimization inconsistency between local linearization training and server-side model reconstruction. This paper uses a first-order approximation to replace the standard bilinear LoRA update, omitting the local interaction terms (δBiδAi\delta B_i\delta A_i). However, server-side aggregation reconstruction introduces new second-order interaction terms. This may lead to discrepancies between the locally optimized model and the final global model.

Questions for authors

1.Compared to directly using local gradients or other gradient initialization strategies, what unique advantages does the use of gradient residuals to select a private adaptation space offer in terms of performance on real-world tasks?

2.Since the private spectral basis is determined only during the initialization phase, how sensitive is this method to changes in the direction of client gradients during subsequent training?

3.How significant are the second-order interaction terms introduced by server-side factor reconstruction in actual training? How do they affect model convergence and final performance?

7 comments
AuthorsOpenReview ↗

Response for Reviewer 9W1N (1/5)

We thank you for the constructive suggestions on private space selection, continued adaptation, and consistency between local optimization and server reconstruction. We organize our responses into two groups: W1, Q1, and Q2 address the necessity of private spaces, residual-based basis selection, and subsequent adaptation; W2 and Q3 address server reconstruction and the practical impact of second-order terms. We explain the design rationale through structural ablations, basis selection comparisons with the same capacity, dynamic refresh experiments, checkpoint interventions, and full training experiments.

W1: The selection of private spaces lacks sufficient validation of necessity. One of the core design elements of this paper is the use of client gradient residuals to select private adaptation spaces. The advantages of this mechanism require further exploration. Furthermore, since the private spectral basis remains frozen during subsequent training, it is difficult to ensure that the selected spaces can continuously adapt to the ever-changing directions of client gradients during the training process.

Q1: Compared to directly using local gradients or other gradient initialization strategies, what unique advantages does the use of gradient residuals to select a private adaptation space offer in terms of performance on real-world tasks?

Q2: Since the private spectral basis is determined only during the initialization phase, how sensitive is this method to changes in the direction of client gradients during subsequent training?

Thank you for suggesting further validation of the contribution of private spaces and their capacity for continued adaptation. We address the necessity of private adaptation, the rationale and evidence for residual basis selection, and continued adaptation within fixed spaces.

A. Necessity of private adaptation

Residual spectral bases allocate private capacity to client-specific directions, while the trainable core UU adapts their combinations to current task gradients. To test private adaptation and basis selection, Table 1(a) compares the full method with independently trained alternatives.

Table 1(a). Ablation of private structures and basis selection. NICO++, 30 clients, shared rank 16/private rank 16, accuracy (%).

Private configurationAutumnDimGrassOutdoorRockWaterAverageCE ↓
Residual gradient bases + full U94.2390.4394.0592.1592.2790.8492.330.3420
Raw local gradient bases + full U94.1290.3594.3491.2891.7390.3292.020.3480
Random orthogonal bases + full U90.9386.5592.6589.6689.6588.3189.630.4220
No private branch (retrained)90.7687.0392.8289.7889.4388.5789.730.4210

The full method improves average accuracy by 2.596 over no private branch, with gains in all six domains, and by 2.703 over random bases with the same UU capacity.

To test whether adaptation is client-specific, we fix the shared AA, BB, and classification head at round 15 and either disable the private branch or swap complete branches between same-domain clients using a fixed mapping.

Table 1(b). Private branch interventions at the same round-15 checkpoint. NICO++, 30 clients, accuracy (%).

Inference settingAutumnDimGrassOutdoorRockWaterAverageCE ↓
Own private branch94.2390.4394.0592.1592.2790.8492.330.3420
Private branch off89.6186.0792.6589.0188.5787.4388.890.4610
Same-domain branch swap87.3082.5990.8886.2886.0883.7886.150.6397

Disabling the branch reduces average accuracy by 3.441; swapping its frozen bases and UU reduces accuracy by 6.177. Thus, the learned private branch contributes to prediction and is matched to individual clients, even within the same domain.

AuthorsOpenReview ↗

Response for Reviewer 9W1N (2/5)

B. Rationale and evidence for residual basis selection

Residual-based basis selection targets the client correction after the common update. At a common reference model, let the client gradient be GiG_i, the weighted average gradient be Gˉ=∑ipiGi\bar G=\sum_i p_iG_i, and the residual be Ri=Gi−GˉR_i=G_i-\bar G. Taking one shared full-gradient step −ηGˉ-\eta\bar G as the reference, a private correction of rank at most rpr_p has the following approximation objective:

\left\\|-\eta\bar G+D_i-(-\eta G_i)\right\\|_F^2 =\min_{\operatorname{rank}(D_i)\le r_p}\\|D_i+\eta R_i\\|_F^2.$$ By the Eckart–Young–Mirsky theorem, exact truncated SVD gives $$D_i^\star=-\eta[R_i]_{r_p}, \qquad \min_{\operatorname{rank}(D_i)\le r_p}\\|D_i+\eta R_i\\|_F^2 =\eta^2\sum_{j > r_p}\sigma_j(R_i)^2.$$ Thus, **the leading residual spectral directions correspond to the optimal client correction under this reference and rank budget**. Raw local gradients contain both common components and client differences; residual-based basis selection concentrates private capacity on corrections relative to the common direction. This result also allows comparison of different spaces with the same dimensions. For any left and right bases with $r_p$ orthogonal columns, denoted by $P,Q$, the optimal residual representation error is $$\mathcal E(P,Q) =\min_C\\|R_i-PCQ^\top\\|_F^2 =\\|R_i\\|_F^2-\\|P^\top R_iQ\\|_F^2.$$ If $P_R,Q_R$ are the leading residual spectral bases, then $$\mathcal E(P,Q)-\mathcal E(P_R,Q_R) =\sum_{j\le r_p}\sigma_j(R_i)^2-\\|P^\top R_iQ\\|_F^2 \ge0.$$ This optimality applies to residual representation at a common reference point. At the same capacity and shared initialization (Table 1(a)), residual bases improve average accuracy over raw local gradient bases by **0.3044**, perform better in five of six domains, and reduce CE from **0.3480** to **0.3420**. To test the benefit at lower private capacity, Table 2(a) compares residual and raw local gradient bases at private rank 4. **Table 2(a). Basis selection comparison with lower private capacity. NICO++, 30 clients, shared rank 16/private rank 4, round 30, accuracy (%).** | Basis source | Autumn | Dim | Grass | Outdoor | Rock | Water | Average | CE ↓ | |---|---:|---:|---:|---:|---:|---:|---:|---:| | Residual gradients | **93.35** | **89.31** | 94.61 | **91.61** | **91.10** | **90.42** | **91.73** | **0.3628** | | Raw local gradients | 93.13 | 89.19 | **94.66** | 91.43 | 90.60 | 90.24 | 91.54 | 0.3671 | At private rank 4, residual bases improve average accuracy by **0.1916**, reduce CE by **0.004330**, and perform better in five of six domains, including a **0.4971** gain on Rock. The benefit therefore persists at lower capacity. To test whether these directions capture client differences across samples, we select bases using 1,024 training images per client and measure residual coverage on 1,024 disjoint training images in DomainNet with 6 clients. All comparisons use the same fixed reference model and private rank 16. **Table 2(b). Residual energy coverage on independent samples. DomainNet, 6 clients, private rank 16.** | Basis source | Residual energy coverage ↑ | |---|---:| | Own residual gradients | **57.21%** | | Raw local gradients | 43.49% | | Population average gradient | 40.47% | | Random orthogonal bases | 0.02990% | Residual bases improve coverage by **13.72** over raw local gradient bases. Together with the task results at equal capacity, this supports their ability to capture reproducible client corrections. This allocation of private capacity complements shared initialization methods such as LoRA-GA and OLoRA, which set the common adapter's starting point.
AuthorsOpenReview ↗

Response for Reviewer 9W1N (3/5)

C. Continued adaptation within fixed spaces

The trainable core UU continually adapts to current gradients, and fixed bases retain predictive value in later training. Absorbing the fixed scaling into the basis matrices, the private weights and their core gradient are

\qquad \nabla_{U_i}f_i=P_i^\top(\nabla_{E_i}f_i)Q_i.$$ With both bases fixed, each update of $U$ uses the current task gradient to recombine directions within the space. Holding the other parameters fixed, let $h_i(U)=f_i(W_{\mathrm{shared}}+P_iUQ_i^\top)$. If $h_i$ is $L_U$-smooth and a full-gradient step $U^+=U-\eta\nabla h_i(U)$ is taken with $0 < \eta\le1/L_U$, then $$h_i(U^+)\le h_i(U)-\frac{\eta}{2}\\|\nabla h_i(U)\\|_F^2.$$ Under these conditions, updating $U$ decreases the objective when the current gradient has a nonzero component in the private space. To test the value of combining directions, Table 3(a) compares independently trained full and diagonal cores. **Table 3(a). Independently trained full versus diagonal cores. NICO++, 30 clients, shared/private rank 16, round 15, accuracy (%).** |Privatecore|Autumn|Dim|Grass|Outdoor|Rock|Water|Average|CE↓| |-|-:|-:|-:|-:|-:|-:|-:|-:| |FullU|**93.73**|**90.71**|**94.17**|91.52|**91.78**|**90.58**|**92.08**|**0.3430**| |DiagonalU|93.46|89.63|93.61|**91.91**|91.19|90.42|91.70|0.3464| The full core improves average accuracy by **0.3793**, performs better in five of six domains, and lowers CE. Both effective private matrices have maximum rank 16, with 6,400 versus 400 trainable core parameters. The comparison supports the benefit of a full core, which offers cross-direction combinations and greater capacity. **Gradient drift and refreshing.** To characterize subsequent gradient drift, let $\mathcal P_i$ be the orthogonal projection onto the initial private space. If the residual gradient is $L_{R,i}$-Lipschitz in the neighborhood under consideration, then $$\\|(I-\mathcal P_i)R_i(W_t)\\|_F \le\\|(I-\mathcal P_i)R_i(W_0)\\|_F +L_{R,i}\\|W_t-W_0\\|.$$ Here, $W$ contains the parameters affecting the gradient. The bound separates initial projection error from displacement-induced gradient drift. To assess the value of updating the private space, we compare fixed bases with periodic refreshes. All groups use the same shared initialization and 128 samples per client for initial basis selection. Each refresh uses the same sample budget and projects the existing private weights into the re-estimated residual space. **Table 3(b). Fixed initial bases versus periodic refreshes. NICO++, 30 clients, shared/private rank 16, round 30, accuracy (%).** |Refreshpolicy|Extrarefreshes|Autumn|Dim|Grass|Outdoor|Rock|Water|Average|CE↓| |-|-:|-:|-:|-:|-:|-:|-:|-:|-:| |Fixedinitialbases|**0**|93.95|**90.27**|94.34|**91.91**|90.96|**90.86**|**92.05**|0.3763| |Refreshevery10rounds|2|93.90|90.03|**94.69**|91.61|**91.28**|90.71|92.04|0.3711| |Refreshevery5rounds|5|**94.39**|89.67|94.64|91.46|**91.28**|90.65|92.02|**0.3703**| Fixed bases achieve **0.01510** and **0.03450** higher average accuracy than refreshes every 10 and 5 rounds, respectively. Refreshes improve some domains and CE. Fixed bases retain the highest endpoint average accuracy without repeated gradient estimation, spectral decomposition, or core transfer. **Predictive contribution later in training.** To test whether fixed private spaces remain useful later in training, we compare inference with and without the private branch at the same NICO++ round-30 and DomainNet round-50 checkpoints. **Table 3(c). Private branch intervention on the fixed-basis model at round 30. NICO++, 30 clients, accuracy (%).** |Inferencesetting|Autumn|Dim|Grass|Outdoor|Rock|Water|Average|CE↓| |-|-:|-:|-:|-:|-:|-:|-:|-:| |Ownprivatebranch|**93.95**|**90.27**|**94.34**|**91.91**|**90.96**|**90.86**|**92.05**|**0.3763**| |Privatebranchoff|89.72|85.91|92.79|89.45|89.43|87.63|89.16|0.4934| At NICO++ round 30, the private branch improves average accuracy by **2.895**, with gains for all 30 clients. The initial space and trained $U$ therefore retain predictive value. **Table3(d).Predictivecontributionofthefixedprivatebranchatround50.DomainNet,30clients,accuracy(%).** |Inferencesetting|Clipart|Infograph|Painting|Quickdraw|Real|Sketch|Average|CE↓| |-|-:|-:|-:|-:|-:|-:|-:|-:| |Ownprivatebranch|**83.84**|**53.51**|**80.09**|**75.63**|**90.39**|**77.74**|**76.87**|**1.275**| |Privatebranchoff|80.65|41.80|74.73|67.11|89.39|71.59|70.88|1.882| At DomainNet round 50, the gain is **5.988**, with improvements in all six domains; Infograph, Quickdraw, and Sketch gain **11.71, 8.520, and 6.154**, respectively, and CE falls from **1.882** to **1.275**. The refresh and checkpoint experiments support **fixed residual bases with a trainable $U$ as a practical balance between adaptation and computational cost in these settings**.
Reviewer_WxwqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors show that initialization can reverse the apparent asymmetry between LoRA factors, and establish that factor agreement alone cannot certify agreement in effective updates. Then, they show that a single gradient step can produce perfectly aligned factors with opposite effective updates, revealing a fundamental limitation of similarity-based interpretations. Motivated by this, they introduce GReP-LoRA, which uses task gradients to guide shared and personalized adaptation. Their method initializes a shared adapter from the population gradient and selects frozen private spectral bases from client residual gradients. Clients jointly learn shared tangent increments and compact private cores that rescale and combine the selected directions, while only shared parameters are aggregated.

Critical strengths
  1. The finding is interesting, i.e., initialization can reverse the apparent asymmetry between LoRA factors.
  2. The motivation of this work is clear.
  3. The proposed method is well empirically motivated.
Critical weaknesses
  1. The private spaces are chosen only at the start. The private bases come from gradients at the starting model and then stay fixed. The authors have shown that the gradients and client models can change later, so the bases may no longer be the best ones. What if we choose the dynamic private spaces?
  2. The results are not good everywhere. With 30 clients, GReP-LoRA is below FedSA-LoRA on DomainNet’s Real and Sketch domains and on NICO++’s Autumn and Dim domains. The method is good on average, but the authors should explain more clearly when the private gradient directions fail and when a simpler private branch may be better.
  3. There exist gaps between the theorem and experiments. The useful improvement results assume a quadratic or locally smooth loss and fixed private spaces, while the actual ViT and GLUE training uses a non-linear model and noisy updates. More results on how the method changes with the distance from the starting model, updated bases, rank, and the amount of client data would show how useful the theory is in practice.
Questions for authors
  1. What are the computation and communication costs of the One-time initialization step? Can it be done without sharing private gradient information?
  2. How sensitive is GReP-LoRA to the number of calibration examples per client and to noise in the estimated gradients?
  3. Can the private bases be updated during training, and is the extra cost worth the improvement?
  4. How does the method perform on text generation, instruction tuning, or partial client participation?
  5. How should users choose the shared rank, private rank, and private core size?
8 comments
AuthorsOpenReview ↗

Response for Reviewer Wxwq (1/6)

Thank you for your careful evaluation of our paper and your specific suggestions regarding the adaptability of private spaces, differences across domains, the scope of the theory, and practical overhead. Following your comments, we have added experiments on dynamic basis selection, classifier head controls, data and calibration budgets, gradient perturbations, text generation, partial client participation, and private core structures. We address your comments and questions below.

W1: The private spaces are chosen only at the start. The private bases come from gradients at the starting model and then stay fixed. The authors have shown that the gradients and client models can change later, so the bases may no longer be the best ones. What if we choose the dynamic private spaces?

Thank you for highlighting the possible changes between initial basis selection and subsequent training. We address this concern through adaptation within a fixed space and the performance and cost of refreshing its bases.

The private bases remain fixed, while the private core UU is continuously trained. The private branch is

\qquad U_i\in\mathbb R^{r_p\times r_p}.$$ The core $U_i$ adapts combinations of directions within the fixed space; representing directions outside it requires changing the bases. To test the benefit of dynamic private spaces, we compare fixed initial bases with refreshing every 10 or 5 rounds. **Table 1(a): Three private space update strategies with the trainable core $U$ retained. NICO++, 30 clients, round 30 endpoint, accuracy (%).** | Setting | Autumn | Dim | Grass | Outdoor | Rock | Water | Average | CE ↓ | |---|---:|---:|---:|---:|---:|---:|---:|---:| | Fixed initial bases | 93.95 | **90.27** | 94.34 | **91.91** | 90.96 | **90.86** | **92.05** | 0.3763 | | Refresh every 10 rounds | 93.90 | 90.03 | **94.69** | 91.61 | **91.28** | 90.71 | 92.04 | 0.3711 | | Refresh every 5 rounds | **94.39** | 89.67 | 94.64 | 91.46 | **91.28** | 90.65 | 92.02 | **0.3703** | **Refreshing improves CE but does not improve average accuracy.** Relative to fixed bases, refreshing every 10 or 5 rounds reduces accuracy by **0.01500** or **0.03440**, while reducing CE from **0.3763** to **0.3711** or **0.3703**. **Fixed bases with a continuously trained $U$ are the default in our experiment**, as they achieve the highest average accuracy and avoid repeated calibration and spectral decomposition.
AuthorsOpenReview ↗

Response for Reviewer Wxwq (2/6)

W2: The results are not good everywhere. With 30 clients, GReP-LoRA is below FedSA-LoRA on DomainNet’s Real and Sketch domains and on NICO++’s Autumn and Dim domains. The method is good on average, but the authors should explain more clearly when the private gradient directions fail and when a simpler private branch may be better.

Thank you for pointing out these differences across domains. We explain domain differences through classifier-head controls and the tradeoff between local adaptation and statistical sharing. The controls compare GReP-LoRA and FedSA-LoRA with shared and private heads on DomainNet and NICO++.

Table 2(a): Classifier head control experiment on DomainNet, 30 clients, accuracy (%). The best result in each column is bold; lower CE is better.

Method / headClipartInfographPaintingQuickdrawRealSketchAverageCE ↓
GReP-LoRA, shared head85.9856.2980.2772.8191.4578.1277.491.057
FedSA-LoRA, shared head82.2350.4378.6572.1491.1076.0575.101.180
GReP-LoRA, private head85.4252.7381.4276.4590.9679.8577.801.202
FedSA-LoRA, private head82.7953.4881.2773.2791.3777.3776.591.263

With shared heads, GReP-LoRA outperforms FedSA-LoRA in all six DomainNet domains, including Real and Sketch, with 77.49% vs. 75.10% average accuracy. A private head gives GReP-LoRA the highest average accuracy (77.80%) and is our choice for DomainNet. It improves Painting, Quickdraw, and Sketch but lowers Clipart, Infograph, and Real accuracy; CE increases from 1.057 to 1.202.

Table 2(b): Classifier head control experiment on NICO++, 30 clients, accuracy (%). The best result in each column is bold; lower CE is better.

Method / headAutumnDimGrassOutdoorRockWaterAverageCE ↓
GReP-LoRA, shared head93.7390.7194.1791.5291.7890.5892.080.3430
GReP-LoRA, private head94.2388.7993.7890.0891.0589.5691.250.4253
FedSA-LoRA, shared head93.9089.3593.4890.7791.1490.0591.450.3760
FedSA-LoRA, private head93.1389.3592.9789.7290.4289.6490.870.4095

On NICO++, GReP-LoRA with a shared head outperforms FedSA-LoRA in five domains and on average (92.08% vs. 91.45%), with Autumn 0.1649 lower. With private heads, GReP-LoRA exceeds FedSA-LoRA by 0.3774, but its own average accuracy falls to 91.25%.

Private heads involve a tradeoff between local adaptation and statistical sharing across clients. A private head can adapt more fully to a client's feature distribution, class proportions, and decision boundaries, but its parameters are estimated mainly from local data, reducing the statistical information that a shared head obtains from other clients. When local samples are limited, class coverage is incomplete, or a domain relies more heavily on discriminative information shared across clients, a private head may more readily fit local biases, increasing estimation variance or weakening generalization. Conversely, a private head may benefit domains that require more adjustment of local decision boundaries.

Domain-level performance depends on the match between private representations and classifier heads, so lower accuracy in a single domain should not automatically be interpreted as a failure of private gradient directions. Calibration error, rank constraints, and directional drift are analyzed in W3 and Q2.

AuthorsOpenReview ↗

Response for Reviewer Wxwq (3/6)

W3: A gap remains between the theory and experiments. The improvement results rely on quadratic or locally smooth losses and fixed private spaces, whereas actual ViT and GLUE training involves nonlinear models and stochastic updates. Further analysis of displacement from initialization, dynamic bases, rank, and client data volume would be helpful.

Thank you for suggesting a closer connection between the theoretical conditions and actual training. We address fixed-space coverage and displacement, client data budgets, and nonconvex optimization with noise and directional bias. Dynamic bases, ranks, and calibration size/noise are addressed in W1, Q5, and Q2, respectively.

Under client-gradient Lipschitz conditions, four factors bound the residual component outside the fixed space:

\\|(I-\mathcal P_i)R_i(W_t)\\|_F \le {}&\operatorname{tail}_{r_p}(R_i(W_0)) +2\varepsilon_i+\delta_{\mathrm{svd},i}\\\\ &+(L_i+\bar L)\\|W_t-W_0\\|. \end{aligned}$$ Here, $\mathcal P_i$ projects orthogonally onto the initial private space; $\operatorname{tail}_{r_p}$ is the initial residual's best rank-$r_p$ approximation error; $\varepsilon_i$ and $\delta_{\mathrm{svd},i}$ are calibration-residual estimation and spectral approximation errors. $L_i$ is the client-gradient Lipschitz constant, $\bar L=\sum_jp_jL_j$, and $p_j$ is the client weight. To test final coverage, we estimate residuals at each strategy's current shared model with the private branch disabled, using training samples disjoint from the 128 private calibration samples. **Table 3(a): Residual coverage on independent samples and displacement of shared target weights at round 30. NICO++, 30 clients, units (%).** | Setting | Initial-basis coverage | Current-basis coverage | Shared weight displacement | |---|---:|---:|---:| | Fixed initial bases | 2.839 | 2.839 | 2.513 | | Refresh every 10 rounds | 2.676 | 5.464 | 2.658 | | Refresh every 5 rounds | 2.588 | 5.672 | 2.679 | *Note: Coverage is the fraction of residual Frobenius energy captured by the two-sided projection, weighted by residual energy. Displacement is the Frobenius norm of the change in the shared adaptation target weights relative to the Frobenius norm of the pretrained target weights.* Refreshing every 10 or 5 rounds raises current-residual coverage from **2.676% to 5.464%** or **2.588% to 5.672%**, respectively. **Higher coverage does not yield higher average accuracy in Table 1(a).** We jointly vary training and calibration budgets from 25% to 100% to test data-volume sensitivity; Q2 isolates calibration. **Table 3(b): Simultaneously varying training and calibration data budgets. NICO++, 30 clients, accuracy (%).** | Setting | Autumn | Dim | Grass | Outdoor | Rock | Water | Average | |---|---:|---:|---:|---:|---:|---:|---:| | 25% training + calibration | 92.91 | 87.55 | 92.15 | 89.25 | 90.06 | 88.18 | 90.02 | | 50% training + calibration | 93.46 | 88.75 | 93.41 | 90.11 | 91.37 | 89.38 | 91.08 | | 100% training + calibration | **93.73** | **90.71** | **94.17** | **91.52** | **91.78** | **90.58** | **92.08** | From 25% to 100% data, average accuracy rises from **90.02% to 92.08%**, with domain gains of about **3.16** on Dim and **0.82** on Autumn. **Conditional nonconvex analysis with fixed private spaces.** Let $x$ collect shared parameters and private cores, and $F(x)=\sum_i p_i f_i(x)$ denote the weighted client objective. Define $\nabla_PF$ and $\\|\cdot\\|_P$ under the client-weighted inner product. The actual update is $$x_{t+1}=x_t-\alpha\bigl(\nabla_PF(x_t)+b_t+\xi_t\bigr), \qquad \mathbb E_t\xi_t=0, \qquad \mathbb E_t\\|\xi_t\\|_P^2\le v^2.$$ Here, $\mathbb E_t$ conditions on the training history; $b_t$ is the conditional mean of the actual update direction minus the target gradient. If $b_t$ has finite second moment, $F$ is bounded below by $F_{\inf}$ and $L_F$-smooth in a neighborhood containing the iterates and segments joining consecutive iterates, and $0 < \alpha\le 1/(4L_F)$, then $$\frac1T\sum_{t=0}^{T-1}\mathbb E\\|\nabla_PF(x_t)\\|_P^2 \le \frac{2(F(x_0)-F_{\inf})}{\alpha T} +L_F\alpha v^2 +\frac4T\sum_{t=0}^{T-1}\mathbb E\\|b_t\\|_P^2.$$ For nonlinear, nonconvex objectives, under conditions including full participation, equal local steps, bounded unbiased ordinary SGD, Lipschitz surrogate gradients, and anchor-point gradient matching, $\\|b_t\\|_P=O(\alpha)$. Over horizon $T$, a fixed step size $\alpha\propto T^{-1/2}$ gives an **$O(T^{-1/2})$** bound on the average squared gradient norm. With momentum, clipping, unequal local steps, or partial participation, the guarantee depends on the explicit bias. We have added these assumptions and the scope of the local quadratic analysis to the appendix.
Reviewer_UZ9POpenReview ↗
Rating
4(+1)
Confidence
3(−1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

GReP-LoRA addresses the limitations of factor similarity in federated LoRA by using task gradients to distinguish shared knowledge from client-specific information. It initializes a shared adapter from population gradients and constructs personalized adapters using private spectral bases derived from residual gradients, enabling selective parameter sharing and personalized adaptation.

Critical strengths
  1. Through initialization-reversal experiments and theoretical counterexamples, the authors demonstrate that factor similarity does not necessarily imply aligned learning directions. This provides a fresh perspective on selective sharing in FedLoRA and serves as the paper's most significant contribution.

  2. The authors propose a comprehensive framework combining average-gradient-based shared initialization, residual-gradient-based private space selection, and locally preserved trainable cores. The method achieves strong results on vision and language tasks, supported by extensive diagnostic experiments and theoretical analyses with clearly stated assumptions.

Critical weaknesses

1.Existing rank sensitivity experiments adjust both shared and private ranks simultaneously, thereby reflecting the impact of changes in overall capacity. However, the authors have not demonstrated the relationship between the size of the private rank and the gains from personalization; an analysis of the optimal ratio between private and shared ranks could address this.

2.Although the appendix provides details on parameter counts and calibration overhead, this information has not been compared against other methods. It is recommended to include a concise resource statistics table that distinguishes between trainable parameters, frozen private parameters, communication volume per round, and one-time initialization overhead, making it easier to evaluate the method's performance and cost.

Questions for authors

1.Although the authors have validated the method using settings with 6 and 30 clients, could they supplement this with experiments involving a larger number of clients (e.g., 100) on a representative dataset to verify the scalability of the personalization benefits?

2.The author states in the paper that a numerically zero residual corresponds to a zero private basis. What criterion is used in the actual implementation—is it that the residual is exactly zero, or simply very small?

5 comments
AuthorsOpenReview ↗

Response for Reviewer UZ9P (1/3)

We thank you for the careful evaluation and constructive suggestions, and for recognizing our analysis of the limitations of factor similarity in federated LoRA. Following your suggestions, we have conducted separate sensitivity analyses for the shared and private ranks, consolidated the resource comparisons, and added an evaluation with 100 clients. We address your comments and questions below.

W1: Existing rank sensitivity experiments adjust both shared and private ranks simultaneously, thereby reflecting the impact of changes in overall capacity. However, the authors have not demonstrated the relationship between the size of the private rank and the gains from personalization; an analysis of the optimal ratio between private and shared ranks could address this.

Thank you for highlighting the importance of distinguishing the effects of shared and private capacity. We examine the separate effects of shared and private rank and the best rank ratios among the tested configurations. On NICO++ with 30 clients, each rank varies while the other remains fixed at 16.

Table 1(a). Varying the shared rank with the private rank fixed at 16. Accuracy (%); the best result in each column is bold.

Shared rankAutumnDimGrassOutdoorRockWaterAverage
493.6890.8393.5690.6291.6490.2691.77
893.9589.9594.2791.4991.9690.0391.94
1693.7390.7194.1791.5291.7890.5892.08
3294.1790.3994.7991.1992.1890.8192.26

Table 1(b). Varying the private rank with the shared rank fixed at 16. Accuracy (%); the best result in each column is bold.

Private rankAutumnDimGrassOutdoorRockWaterAverage
492.9688.6393.7390.9590.2889.6491.03
893.3590.1194.1291.6191.6090.1191.82
1693.7390.7194.1791.5291.7890.5892.08
3294.6190.6394.0090.8691.5090.4792.01

Shared and private ranks affect performance differently:

  • Private rank: At shared rank 16, increasing private rank from 4 to 16 raises average accuracy from 91.03% to 92.08%, a gain of 1.050; rank 32 yields 92.01%. Gains saturate around private rank 16.
  • Shared rank: At private rank 16, increasing shared rank from 4 to 32 raises average accuracy from 91.77% to 92.26%, a gain of 0.4900.

The highest tested average occurs at (shared rank, private rank) = (32, 16), a shared-to-private ratio of 2:1. At shared rank 16, private rank 16 performs best (1:1). These ratios apply to their respective tested configurations. The default (16, 16) yields 92.08% average accuracy, 0.1800 below the highest tested average. We have added both experiments and clarified this scope in the revised manuscript.

AuthorsOpenReview ↗

Response for Reviewer UZ9P (2/3)

W2: Although the appendix provides details on parameter counts and calibration overhead, these have not been compared with other methods. A concise resource statistics table distinguishing trainable parameters, frozen private parameters, communication volume per round, and one-time initialization overhead would make it easier to evaluate the method's performance and cost.

Thank you for suggesting a consolidated resource comparison. We address resource costs and the tradeoff between communication and performance through comparisons of parameters, communication, computation, and GPU memory, including bidirectional SVD compression.

Table 2(a). Unified resource comparison using the same target layers and rank setting.

MetricFedITFedSA-LoRALoRA-FAIRGReP-LoRA (uncompressed)GReP-LoRA (bidirectional SVD, r=16)
Total trainable parameters (M)1.1031.1031.1031.1091.109
Private trainable parameters (K)0353.306.4006.400
Additional frozen private parameters (M)0001.0571.057
Routine bidirectional communication per round (MiB/client)33.6622.8833.6633.6633.66
One-time communication cost (MiB/client)000274.533.66
Average communication including one-time cost, amortized over 100 rounds (MiB/client/round)33.6622.8833.6636.4134.00
Local update time per 128 samples (s)0.45300.45170.45180.54450.5445
Server computation per round (s)0.049800.0357087.200.048600.04860
Peak allocated GPU memory (MiB)80148014801480858085
Peak reserved GPU memory (MiB)81388138818483808380
One-time preparation and initialization (s)1.1401.3201.30092.9092.90
Training compute budget (h)6.4036.3857.1127.6977.697
Total compute cost including initialization (h)6.4036.3857.1137.7237.723

Relative to FedIT, both configurations add 6.400 K trainable parameters (0.5803%) and retain 1.057 M frozen private parameters locally. Routine communication matches FedIT and LoRA-FAIR; FedSA-LoRA communicates less. Including initialization raises uncompressed GReP-LoRA's average communication from 33.66 to 36.41 MiB/client/round over 100 rounds.

Uncompressed GReP-LoRA increases local update time by 20.20% and peak allocated GPU memory by 0.8814% over FedIT. Preparation and initialization take 92.90 s. Including initialization, the 30-round total computation time is 7.723 h, versus 6.403 h for FedIT.

To reduce communication costs and direct exposure of full matrices, we transmit truncated SVD representations with rank 16 in both directions. Table 2(a) shows routine communication of 33.66 MiB/client, matching FedIT. Over 100 rounds, amortized communication falls from 36.41 MiB/client/round uncompressed to 34.00 MiB/client/round. To measure the performance cost, we compare transmission schemes on NICO++ with 30 clients over 30 rounds.

Table 2(b). Performance comparison between uncompressed transmission and bidirectional SVD with rank 16. NICO++, 30 clients, six-domain accuracy (%) at round 30. The highest accuracy or lowest CE in each column is bold, including ties.

MethodAutumnDimGrassOutdoorRockWaterAverageCE ↓
GReP-LoRA, uncompressed94.6790.7594.4991.4092.0090.7692.350.3514
GReP-LoRA, bidirectional SVD, r=1694.1790.7594.4791.5591.9190.7692.270.3587

Rank 16 bidirectional SVD yields 92.27% average accuracy versus 92.35% uncompressed, a decrease of 0.07670 calculated from unrounded values; CE rises from 0.3514 to 0.3587. It reduces amortized communication with a small accuracy loss. We have added the resource table, transmission procedure, and performance comparison to the revised manuscript.

AuthorsOpenReview ↗

Response for Reviewer UZ9P (3/3)

Q1: The authors have evaluated the method with 6 and 30 clients. Could they add an experiment with more clients (e.g., 100) on a representative dataset to verify whether the personalization benefits persist at a larger client scale?

Thank you for suggesting a further evaluation of personalization benefits with more clients. We evaluate scalability through domain-wise accuracy, average gains over baselines, and CE on NICO++ with 100 clients, comparing all methods at a common round-30 endpoint.

Table 3. NICO++ results with 100 clients at round 30. Accuracy (%); the highest accuracy or lowest CE in each column is bold.

MethodAutumnDimGrassOutdoorRockWaterAverageCE ↓
GReP-LoRA92.8590.4793.8592.0091.6990.6391.920.3889
FedIT90.8787.0793.2689.6989.8389.0989.970.4025
FedSA-LoRA90.2186.7191.9188.9288.5287.2788.920.4561
LoRA-FAIR89.9985.5992.6088.3288.4888.2888.880.4481
FFA-LoRA90.3886.5592.4789.5789.8389.3889.700.3941

GReP-LoRA achieves the highest accuracy in all six domains, 91.92% average accuracy, and the lowest CE (0.3889). Average gains over FedIT, FedSA-LoRA, LoRA-FAIR, and FFA-LoRA are 1.945, 2.992, 3.039, and 2.217, respectively, supporting personalization benefits in this 100-client setting. We have added this experiment to the revised manuscript.

Q2: The paper states that a numerically zero residual corresponds to a zero private basis. What criterion is used in the implementation: must the residual be exactly zero, or is a sufficiently small residual treated as zero?

Thank you for requesting clarification of this implementation detail. We clarify the threshold, its effect on each private branch, and whether it is triggered in the calibration libraries. We compute the Frobenius norm of the residual matrix in FP64 using this absolute tolerance:

∥Ri∥F≤10−12.\|R_i\|_F \leq 10^{-12}.

Residuals meeting this threshold are treated as numerically zero: we set the private basis for the corresponding client and layer to zero and disable that private branch.

The full-data and 128-sample calibration basis libraries each contain 750 client-layer entries, with all 750 entries active in each library. Their minimum residual norms, 0.5184 and 1.087, respectively, exceed the threshold, so all entries retain their private adaptation spaces. We have specified the criterion and computation precision in the revised manuscript.

Thank you again for your constructive suggestions. Could you please let us know whether these additional experiments and clarifications address your questions and concerns? We would be happy to continue the discussion and provide further details in response to your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief summary of all responses

Dear AC and Reviewers,

We sincerely thank you for your careful review and constructive suggestions. In response to the reviewers’ concerns, we have added the following experiments and clarifications:

  • Private-space contribution and residual-based selection (9W1N): We added structural ablations, private-branch interventions, and low-rank basis comparisons in Appendix H.1, Tables 4–6.
  • Rank and core structure (UZ9P, Wxwq): We separately varied the shared and private ranks and compared full and diagonal cores in Appendix H.2, Tables 7–9.
  • Fixed versus dynamic private spaces (Wxwq, 9W1N): We evaluated periodic basis refresh, residual coverage, model displacement, later-checkpoint interventions, and refresh costs in Appendix H.3, Tables 10–13.
  • Domain-level variation (Wxwq): We added shared/private classifier-head controls on DomainNet and NICO++ in Appendix H.4, Tables 14–15.
  • Calibration, data budgets, and gradient noise (Wxwq): We added experiments varying calibration size, client data budgets, and gradient perturbations in Appendix H.5, Tables 16–18.
  • Numerical degeneracy criterion (UZ9P): We specified the implementation criterion for numerically zero residuals and documented calibration-bank checks in Appendix H.6.
  • Resource costs and initialization transmission (UZ9P, Wxwq): We added parameter, communication, computation, and memory comparisons, evaluated SVD transmission, and clarified its privacy implications in Appendix H.7, Tables 19–21.
  • Scalability (UZ9P) and partial participation (Wxwq): We added experiments with 100 clients and different participation ratios, respectively, in Appendix H.8, Tables 22–23.
  • Instruction tuning and text generation (Wxwq): We added Dolly-15k and DialogSum evaluations, including an evaluation after removing overlapping test inputs, in Appendix H.8, Tables 24–26.
  • Server reconstruction and second-order effects (9W1N): We added geometric diagnostics, checkpoint interventions, and a full-training ablation in Appendix H.9, Tables 27–30.
  • Connections between theory and actual training (Wxwq, 9W1N): We clarified the theoretical assumptions, optimization-error terms, and their relation to the training mechanisms in Appendices F, L, and O.

We have also clarified the averaging and standard-deviation definitions in Tables 2–3. All additions and revisions are highlighted in blue.

Please let us know whether these revisions address your concerns. We would be happy to discuss any remaining questions. Thank you again for your time and consideration.

Best regards,
The Authors of Paper 59744

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.