acceptodds
Under review as a conference paper at ICLR 2027

FedCSO-LoRA: Coordinated Subspace Optimization for Heterogeneous Federated LoRA

Abstract

Low-Rank Adaptation (LoRA) enables an efficient federated fine-tuning of large language models. In practice, however, clients with heterogeneous resources may have different rank budgets, constraining the dimensions of their active latent subspaces. Some heterogeneous-rank methods randomly select the active components of a shared LoRA adapter, which may miss the informative combinations of components that could be learned even with the same rank budget. Learning subspaces from the local gradients can capture these combinations, but independently selected client subspaces may yield poorly aligned updates when data are heterogeneous. The challenge is therefore to preserve locally the informative directions under different rank budgets, while coordinating the resulting client updates towards the global objective. To this end, we propose FedCSO-LoRA, a coordinated subspace optimization framework for heterogeneous federated LoRA. Given its rank budget, each client learns the combinations of shared adapter coordinates that maximize the retained energy of its estimated local factor gradients. The coordination step aligns the selected subspace with a global update reference, while preserving a prescribed fraction of the initially retained gradient energy. The server maps the client updates into the common factor coordinates, and filters their aggregate according to the subspace coverage. We provide a nonconvex convergence analysis under the heterogeneous rank budgets that characterizes the effects of local training and server aggregation. Experiments across multiple datasets and language models demonstrate the effectiveness of FedCSO-LoRA compared with the various baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
federated learning, low-rank adaptation, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nwnAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies federated LoRA for heterogeneous devices: different devices can support different LoRA ranks, and this paper optimizes a projection matrix Q_i to adjust the dimension of B_i and A_i (forming BQ and Q^T*A). To optimize the matrix Q_i, this paper mainly considers 2 directions: (1) Q_i should maintain enough "full gradient" energy (Eq. 4), and (2) the projection Q_i should make sure the client subspaces are aligned (Eq. 7-9). Their solution is motivated well, with theoretical and experimental results to support the effectiveness of the algorithm.

Critical strengths

The most of the paper is clear to understand and authors unfold their main algorithm (section 4.1 and 4.2) in a good structure, the problem is meaningful for the topic of federated LoRA. I agree on authors' idea to optimize Q for both maintaining gradient energy and subspace alignment, from my knowledge of this field, this is a clear and well motivated solution, so the good experimental results are reasonable.

Critical weaknesses
  1. Figure 1 and 2 should be improved. I don't get the point of Figure 1 (c). For figure 1 (a), it would be helpful to clarify the dimension of B, Q in the figure. For Figure 2, step 2 seems conflict with the algorithm you described. In Eq. 9, your optimization is one-step, while in figure 2 step 2, you separate this into two optimization steps (local-coordinated). Also in Figure 2, place specify what these rectangles are, B, A, Q?
  2. The algorithm seems to be expensive as it requires one round of gradient computation in the large rank R space (contradict with the assumption that the device cannot afford the training of rank R), and also the coefficient c is decided by a greedy search (is this search per-round or one-time for the whole training?).
  3. Vague motivation for the exploration (line 273 to 276). If you solve Eq. 9, you get the optimal solution in terms of your objective, what is the aim of this exploration and sampling?
  4. Authors defined some unnecessary and redundant notations that actually stop human readers to understand the paper, like P=QQ^T.
Questions for authors
  1. Please explain the motivation for using the exploration (line 273 to 276).
  2. Can authors provide clearer explanation for the cost of FedCSO-LoRA? Theoretical complexity or more detailed experimental metrics other than GPU hour?
  3. One recent work FedRot-LoRA also optimizes an orthonormal basis Q_i, what's the difference between their optimization and your algorithm? I feel they mostly focus on the subspace alignment (like your section 4.2), is FedRot-LoRA a complementary method for your setting or you may want to compare it in the experiments?

I think the paper's idea is meaningful and good, while many issues should be addressed before the paper reaches the acceptance bar.

7 comments

Thank you for the detailed feedback. We have revised Figures 1 and 2, clarified why we use exploration after solving Eq. (9), and added measurements of peak GPU memory and runtime.

W1: Figures 1 and 2

Comment: Figure 1 and 2 should be improved. I don't get the point of Figure 1 (c). For figure 1 (a), it would be helpful to clarify the dimension of B, Q in the figure. For Figure 2, step 2 seems conflict with the algorithm you described. In Eq. 9, your optimization is one-step, while in figure 2 step 2, you separate this into two optimization steps (local-coordinated). Also in Figure 2, place specify what these rectangles are, B, A, Q?

Response:

We have revised Figures 1 and 2 and their captions as follows.

1). Figure 1 (a) now labels BB, QiQ_i, and BQiBQ_i with dimensions m×Rm\times R, R×kiR\times k_i, and m×kim\times k_i, respectively. Both rows illustrate R=6R=6 and ki=2k_i=2.

2). The revised caption of Figure 1 (c) defines the horizontal axis as the cosine similarity between the projected local gradient and the previous global update reference. The vertical axis shows accuracy changes in percentage points. On both tasks, accuracy first improves and then declines as alignment increases. This trend motivates limiting the loss of local gradient information during coordination.

3). Equation (9) constructs each candidate basis using local gradient information and the global reference together. The local-only case c=0c=0 provides the energy baseline for the acceptance conditions in Eq. (10). We have therefore redrawn Step 2 to show both inputs leading to preferred basis construction, followed by subspace sampling. This replaces the potentially misleading “Local → Coordinated” sequence.

4). The revised caption of Figure 2 identifies the vertical and horizontal rectangles as BtB^t and AtA^t in the global adapter, and Bˉi\bar B_i and Aˉi\bar A_i in Step 3. In Step 2, Qi⋆Q_i^\star denotes the preferred basis and QiQ_i denotes the sampled training basis.

W2: Full-rank gradient estimation and search frequency

Comment: The algorithm seems to be expensive as it requires one round of gradient computation in the large rank R space (contradict with the assumption that the device cannot afford the training of rank R), and also the coefficient c is decided by a greedy search (is this search per-round or one-time for the whole training?).

Response:

We estimate gradients and select cc at round start, then keep the resulting subspace fixed throughout local training. Rank-wise block estimation and layer-wise offloading reduce GPU memory requirements, while the search for cc uses precomputed matrices without additional backbone forward or backward passes. Our measurements show modest overall runtime overhead.

1). Rank-RR gradient estimation has modest overhead.

We use one mini-batch to estimate the rank-RR factor gradients for subspace selection. This computation is performed at round start and is not repeated at every local step. In the manuscript’s benchmark configuration, the full method uses 11.3 GPU hours on GLUE and 45.8 GPU hours on commonsense reasoning, compared with 10.9 and 44.3 GPU hours for FSLoRA. These increases of approximately 3.7% and 3.4% include the cost of gradient estimation and subspace construction.

To accommodate clients with limited GPU memory, we provide two strategies.

(a) Rank-wise block estimation. Computing the gradients over all RR directions does not require all of them to be active simultaneously on the GPU. As described in Appendix A.3, each backward pass computes gradients for at most kik_i active directions, with the remaining directions folded into the frozen weights. All blocks use the same round-start model and estimation batch. Completed gradient blocks are moved to host memory, avoiding the need to allocate the complete rank-RR factor gradients on the GPU.

(b) Layer-wise backbone offloading. For tighter GPU memory budgets, we also evaluated layer-wise BF16 offloading. In the one-local-step benchmark in Table C1, a gradient-estimation microbatch size of 4 reduces peak allocated GPU memory by 64.24%, with a 12.60% increase in total runtime. A microbatch size of 2 further reduces memory by 76.67%, at a higher runtime cost. Here, microbatch size denotes the number of examples processed together during gradient estimation.

Table C1. Peak PyTorch allocated GPU memory and runtime for a short run with one local step on the LLaMA-3.2-3B commonsense reasoning client.

ConfigurationGradient-estimation microbatch sizeGradient estimation timeTotal runtime with one local stepPeak allocated GPU memory
GPU-resident BF16 reference13.07 s11.98 s7,435.64 MiB
Layer-wise BF16 offloading24.83 s15.40 s1,735.02 MiB
Layer-wise BF16 offloading42.71 s13.49 s2,659.16 MiB

In a separate FP32 GLUE experiment, layer-wise offloading achieved the same accuracy as the GPU-resident configuration at the reported precision.

2). The coordination search has low overhead.

The search for cc uses precomputed R×RR\times R matrices and bounded expansion and bisection when needed. Testing candidate values requires no additional backbone forward or backward passes. In our MRPC benchmark with 20 participating clients, the search across all clients and layers took 2.52 s per round, about 3.1% of the median round time of 81.25 s.

In GLUE, each participating client runs this search once per layer at the start of each round from round 2 onward. The first round uses local gradient selection because no historical reference is available. The resulting training subspace remains fixed for all 50 local SGD steps.

W3 / Q1: Motivation for exploration

Comment (W3): Vague motivation for the exploration (line 273 to 276). If you solve Eq. 9, you get the optimal solution in terms of your objective, what is the aim of this exploration and sampling?

Comment (Q1): Please explain the motivation for using the exploration (line 273 to 276).

Response:

Equation (9) of the original manuscript uses the estimated local gradient and the global reference to select a preferred subspace within the client’s rank budget. Even when this selection is optimal, directions outside the subspace may still contain gradient information.

1). Exploration allows updates outside the preferred subspace.

Exploration gives these remaining directions a chance to contribute, while each sampled subspace still contains only kik_i directions. We then weight each selected direction by the inverse of its inclusion probability. As shown in Eq. (43) of the original manuscript, the corrected estimate recovers the full local factor gradient in expectation over direction sampling at round start. Without exploration, gradient components outside the preferred subspace would be excluded and could not be recovered by this correction.

2). Energy-guided selection controls sampling variance.

Correction preserves the mean gradient, but the sampling probabilities still determine how often each direction contributes and how much it is weighted. Preferred directions are selected more often and receive smaller correction weights. For a fixed gradient and fixed inclusion probabilities, Eq. (44) of the original manuscript shows that placing more gradient energy in these frequently sampled directions reduces the variance of the corrected estimate. Thus, exploration and correction preserve the full gradient in expectation, while energy-guided selection helps reduce the expected squared error for the gradient used to construct the basis.

W4: Notation

Comment: Authors defined some unnecessary and redundant notations that actually stop human readers to understand the paper, like P=QQ^T.

Response:

We will simplify the notation in the main text by replacing PiP_i with QiQi⊤Q_iQ_i^\top, avoiding a separate symbol for a quantity already determined by QiQ_i. For example, Eq. (11) of the original manuscript will read W0,ifold=W0+B(I−QiQi⊤)AW_{0,i}^{\mathrm{fold}}=W_0+B(I-Q_iQ_i^\top)A. We will also use consistent symbols for the factors and bases across the text and figures.

Q2: Memory and runtime costs

Comment: Can authors provide clearer explanation for the cost of FedCSO-LoRA? Theoretical complexity or more detailed experimental metrics other than GPU hour?

Response:

To clarify the cost beyond GPU hours, we compare FSLoRA and FedCSO-LoRA stage by stage on MRPC and report peak GPU memory for FedCSO-LoRA. We also provide a separate experiment showing the memory and runtime trade-off of offloading.

1). Complete-round runtime and memory.

Our MRPC benchmark uses 20 participating clients per round and 50 local training steps per client. As shown in Table C2, the median round time is 81.25 s for FedCSO-LoRA and approximately 75.55 s for FSLoRA, an increase of about 5.70 s. Local training accounts for most of the runtime in both methods, taking 76.73 s for FedCSO-LoRA and 75.55 s for FSLoRA. Among the additional stages in FedCSO-LoRA, coordination search takes 2.52 s, the largest stage cost outside local training. FedCSO-LoRA's peak PyTorch allocated GPU memory is 2,368.62 MiB.

Table C2. Median stage and complete-round runtimes for FSLoRA and FedCSO-LoRA on MRPC.

StageFSLoRA (s)FedCSO-LoRA (s)
Gradient estimation—1.40
Basis construction—0.34
Coordination search—2.52
Exploration—0.18
Local training75.5576.73
Server update0.0020.075
Complete round75.5581.25

Stage medians and complete-round medians are computed separately. A dash indicates that the stage is absent in FSLoRA.

2). Memory and runtime with offloading.

For clients with tighter GPU memory budgets, we evaluate layer-wise BF16 offloading on LLaMA-3.2-3B for commonsense reasoning. This separate benchmark includes gradient estimation and one local training step.

As reported in Table C1 of our response to W2, the offloaded configuration with a gradient-estimation microbatch size of 4 reduces peak allocated GPU memory from 7,435.64 MiB to 2,659.16 MiB, while runtime increases from 11.98 s to 13.49 s. This corresponds to 64.24% less GPU memory and 12.60% longer runtime in this benchmark. A microbatch size of 2 further reduces the peak to 1,735.02 MiB, with a runtime of 15.40 s, providing another option when memory is more limited.

We will add these measurements and their configurations to the revised manuscript.

Reviewer_syQ5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FedCSO-LoRA for federated training where clients have different adapter rank budgets. It uses local gradients and the previous global update to choose which parts of a shared adapter each client trains, while preserving the full global model at initialization. The server then combines client updates and downweights weakly covered directions. The paper provides theoretical results on gradient preservation and convergence, and reports accuracy gains on GLUE and commonsense reasoning tasks, with ablations supporting its main components.

Critical strengths
  • The gradient-based subspace selection is simple and principled. Under a fixed client rank budget, it identifies combinations of adapter components that preserve the most estimated local gradient energy. This gives a clear mathematical reason for choosing which directions to train, beyond randomly selecting components or ranking them individually.
  • The experiments provide meaningful support for the design. Beyond improvements on GLUE and commonsense reasoning tasks, the paper compares learned subspaces against both random and importance-based coordinate selection and separately ablates coordination and filtering. These controls help establish that the proposed components contribute to the observed gains.
Critical weaknesses
  • The reliability of the local subspace initialization is insufficiently examined. The method uses one batch to select the active adapter factors, then holds the selected basis fixed throughout local training. Although this preserves the initial global model, it changes the trainable parameterization and subsequent optimization. The energy-maximization result applies to the estimated initial gradient; it does not establish that the selected directions remain useful across other batches or later training steps. The paper does not directly examine this through estimation-batch-size sensitivity, refresh-frequency studies, or measurements of retained gradient energy during training. This leaves an important gap between the central selection principle and its claimed training benefit.
  • The gains from coordination and coverage filtering needs more explanations. The ablations support their usefulness, but the theory does not establish why the historical update is a reliable guide or why weakly covered directions should be attenuated. The coordination result concerns agreement with a historical reference before probability correction, while the filtering result gives a conditional bias–variance trade-off without showing that coverage predicts reliability. The main convergence theorem also applies to arbitrary preferred bases under its assumptions. These limitations matter because the additional components contribute substantially to the reported gains, yet their benefits remain only partially explained.
Questions for authors
  1. How are the initial global factors (B_0,A_0) initialized? Have you examined whether the benefit of gradient-based subspace selection depends on this initialization?

  2. Does the selected subspace retain more gradient energy on independent local batches, beyond the batch used to select it? How sensitive are the results to the estimation batch size?

  3. How frequently is the preferred basis refreshed in the reported experiments? Does its gradient-energy advantage persist throughout the local training steps?

  4. Is there evidence that coordination improves agreement with the current global gradient or reduces conflict between actual client updates after local training, beyond improving agreement with the historical reference?

  5. Do weakly covered directions exhibit greater update noise or estimation error in practice? Can the filtering benefit be distinguished from simply reducing the server learning rate or applying uniform shrinkage?

  6. The vanishing-error convergence result requires the filter strength to decrease with the learning rate, whereas the experiments use a fixed target strength. What theoretical or empirical evidence supports the experimental schedule?

  7. Does the implementation satisfy the theorem’s requirements of plain SGD, training batches independent of basis construction, and fresh direction sampling each round? If there are deviations, which reported results remain covered by the analysis?

  8. In the FLoRA comparison, can the accumulated update relative to the pretrained model grow beyond the proposed method’s fixed global rank? How should this difference in model capacity and adapter reinitialization affect our interpretation of the comparison?

5 comments

Thank you for the detailed feedback. We measured how much gradient energy the selected subspaces retain on independent batches and over 50 local training steps. We also tested whether coordination reduces conflicts between client updates and compared coverage filtering with uniform shrinkage at the same update norm.

W1 / Q2: Independent batches and estimation batch size

Comment (W1): The reliability of the local subspace initialization is insufficiently examined. The method uses one batch to select the active adapter factors, then holds the selected basis fixed throughout local training. Although this preserves the initial global model, it changes the trainable parameterization and subsequent optimization. The energy-maximization result applies to the estimated initial gradient; it does not establish that the selected directions remain useful across other batches or later training steps. The paper does not directly examine this through estimation-batch-size sensitivity, refresh-frequency studies, or measurements of retained gradient energy during training. This leaves an important gap between the central selection principle and its claimed training benefit.

Comment (Q2): Does the selected subspace retain more gradient energy on independent local batches, beyond the batch used to select it? How sensitive are the results to the estimation batch size?

Response:

Following the reviewer’s suggestion, we evaluate the selected bases on independent batches and vary the estimation batch size. The bases retain more energy than a same-rank random basis, while retention and final accuracy remain similar with 16, 32, or 64 estimation examples. We report the local-training and refresh-frequency studies in our response to Q3.

1). The selected bases retain more energy on independent batches.

We construct the bases using one local batch, then keep them fixed and evaluate their energy retention on independent local batches. We compare local gradient-based selection, coordinated selection, and a random orthonormal basis of the same rank.

For each evaluation batch, we report the captured energy as a percentage of the maximum energy achievable at the same rank on that batch. Thus, the comparison measures how closely the selected bases approach the best subspace for an independent batch.

Table B1. Gradient energy retention on independent local batches, relative to the maximum achievable at the same rank.

TaskLocal gradient-based basisCoordinated basisSame-rank random orthonormal basis
MRPC91.95%87.21%51.94%
CoLA89.57%85.49%52.78%
RTE89.20%85.11%52.96%

Both selected bases retain substantially more energy than the random basis. These results support that the selected directions remain informative beyond the batch used to construct them.

2). The results are stable across the tested estimation batch sizes.

Across all seven GLUE tasks, we use 16, 32, or 64 examples for basis construction and measure energy retention on independent local batches, together with final mean accuracy.

Table B2. Energy retention and final accuracy by estimation batch size.

Examples used for basis estimationMedian energy retentionFinal mean GLUE accuracy
1695.15%85.6%
3295.14%85.6%
6495.11%85.6%

Increasing the estimation batch size produces little change in the reported retention, and final mean accuracy is the same at the reported precision. This supports using 16 examples as the default in the tested setting.

To address whether the basis remains useful during subsequent training, our response to Q3 provides the local-step and refresh-frequency studies. Median energy retention decreases from 94.87% at step 0 to 91.57% at step 50. Once-per-round refresh and per-step re-estimation achieve mean GLUE accuracies of 85.6% and 85.8%, respectively. These results provide additional evidence for keeping the basis fixed within a round.

We will include these experiments in the revised manuscript to clarify the practical support for the gradient-based selection criterion.

Q1: Initial factors and initialization dependence

Comment: How are the initial global factors (B_0,A_0) initialized? Have you examined whether the benefit of gradient-based subspace selection depends on this initialization?

Response:

We initialize B0B_0 to zero and A0A_0 randomly using Kaiming uniform initialization. Following this suggestion, we also test three different initializations of A0A_0. FedCSO-LoRA achieves higher mean GLUE accuracy than FSLoRA [B1] in all three cases, showing that its advantage is not limited to the default initialization.

With B0=0B_0=0, the initial adapter contribution B0A0B_0A_0 is zero, so training starts from the pretrained model. To examine the effect of the random initial values, we test the default initialization and two additional random initializations of A0A_0 from the same Kaiming uniform distribution. In each case, B0B_0 remains zero. Table B3 reports each method's mean accuracy across the seven GLUE tasks.

Table B3. Mean GLUE accuracy across three initializations of A0A_0.

InitializationFSLoRAFedCSO-LoRA
Default83.3%85.6%
Initialization 283.1%85.3%
Initialization 383.0%85.8%
Mean across initializations83.1%85.6%

FedCSO-LoRA improves mean accuracy by 2.2–2.8 percentage points across these initializations. We will specify the initialization procedure and add this comparison to the revised manuscript.

[B1] Fang et al. Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs. ICML 2026.

Q3: Refresh frequency and within-round persistence

Comment: How frequently is the preferred basis refreshed in the reported experiments? Does its gradient-energy advantage persist throughout the local training steps?

Response:

To clarify our implementation, we refresh the preferred basis at the start of each communication round, then sample the training subspace and keep it fixed for all 50 local steps. Our additional experiments support this choice: the round-start basis retains high gradient energy during training, while per-step re-estimation provides little accuracy improvement at a higher measured cost.

1). Gradient energy retention remains high during local training.

We keep the round-start basis fixed and measure how much gradient energy it captures at subsequent local iterates. As shown in Table B4, median retention decreases by 3.30 percentage points over 50 steps, from 94.87% to 91.57%.

Table B4. Gradient energy retained by the fixed round-start basis during local training.

Local step0151050
Median gradient energy retained94.87%94.79%94.13%93.64%91.57%

The gradients also remain closely aligned between steps 0 and 10. Their median cosine similarities are 0.996 on MRPC, 0.999 on CoLA, and 0.998 on RTE. These measurements support that the selected directions remain useful as local training proceeds.

2). Per-step re-estimation adds cost with little accuracy improvement.

We compare the default once-per-round refresh with per-step re-estimation across all seven GLUE tasks. As shown in Table B5, final mean accuracy increases only slightly, from 85.6% to 85.8%.

Table B5. Final GLUE accuracy by basis refresh frequency.

GLUE taskOnce-per-round refreshPer-step re-estimation
MRPC89.3%89.5%
CoLA83.8%83.9%
RTE75.1%75.5%
SST-294.1%94.3%
QNLI89.7%89.7%
MNLI-m80.9%81.0%
QQP86.3%86.4%
Task mean85.6%85.8%

In a separate timing experiment with 10 local steps per round, per-step re-estimation took 2.72–2.89 times as long per round as once-per-round refresh. The small accuracy gain and higher runtime support reusing the basis within each round.

We will specify the refresh schedule and include these measurements in the revised manuscript.

W2 / Q4: Current-gradient alignment and actual update conflicts

Comment (W2): The gains from coordination and coverage filtering needs more explanations. The ablations support their usefulness, but the theory does not establish why the historical update is a reliable guide or why weakly covered directions should be attenuated. The coordination result concerns agreement with a historical reference before probability correction, while the filtering result gives a conditional bias–variance trade-off without showing that coverage predicts reliability. The main convergence theorem also applies to arbitrary preferred bases under its assumptions. These limitations matter because the additional components contribute substantially to the reported gains, yet their benefits remain only partially explained.

Comment (Q4): Is there evidence that coordination improves agreement with the current global gradient or reduces conflict between actual client updates after local training, beyond improving agreement with the historical reference?

Response:

To address this concern, we evaluate the actual updates after probability correction and local training. Coordination reduces conflicts between client updates and improves their aggregate alignment with the current estimated descent direction on average. These results support its practical benefit beyond agreement with the historical reference used for subspace selection.

Specifically, we express client updates in shared LoRA-factor coordinates and count pairs with negative cosine similarity. We also measure the cosine similarity between the aggregate update before server filtering and the current estimated descent direction. The current global gradient is estimated at the shared round-start model by averaging factor gradients over 20 clients, using 32 held-out examples per client.

Table B6. Client update conflicts and alignment with the current estimated descent direction.

TaskConflicts without coordinationConflicts with coordinationAlignment without coordinationAlignment with coordination
MRPC100/19095/1900.3510.381
CoLA92/19089/1900.4180.440
RTE105/19098/1900.2960.342
SST-296/19096/1900.5620.562
QNLI101/190103/1900.4730.466
MNLI-m108/190102/1900.3910.425
QQP94/19092/1900.5140.526
Total conflicts / mean alignment696/1,330675/1,3300.4290.449

Mean alignment increases from 0.429 to 0.449, while the conflict rate decreases by 1.58 percentage points. Both metrics improve on five tasks, remain unchanged on SST-2, and worsen slightly on QNLI.

Theorem 5.1 of the original manuscript provides a general convergence guarantee, while these diagnostics examine the benefit of the specific coordination strategy. We will include these measurements in the revised manuscript and address coverage filtering separately in our response to Q5.

Reviewer_H2c7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses federated LoRA finetuning when clients have heterogeneous rank budgets. prior methods use to train a subset of the existing adapter coordinates. here, each client selects an orthonormal basis, a linear combination of coordinates, that maximizes the retained energy of locally estimated factor gradients. this basis is then refined toward agreement with a shared reference. The previous global update recovered from cached factors, while retaining at least a fraction of the locally optimal energy. clients train in a folded parameterization that preserves the full global model, with exploration and inverse probability gradient correction. The server lifts client update back to global coordinates and applies a bounded coverage filter.

Critical strengths
  • authors understands that each client may have some local best direction but this best not necessarily be best for global model . therefore they tied it with global coordination.

  • clients trained on low rank lora then also in experiments shown imporvements wrt baselines. more useful learning in same budget, with combinations of directions while keeping the same number of trainable parameters.

Critical weaknesses
  • the paper selects directions that capture maximum estimated gradient energy. Proposition 4.1 showes this choice is optimal for energy.equation 31 links energy to loss decrease, but only for a single exact , uncorrected gradient step. Actual method uses a one-batch gradient estimate, proability corrected sample subspaces, and many local steps under non IID data, so this guarantee does not carry over. table 8 also shows that the cordinated basis retains less energy than the local basis bit achives higher accuracy. .so, why is gradient energy is the right proxy for the optimization quality, especially under non-IID client data.

  • paper stated problem is clients with different capabilities. but in convergence proof authors assumed equal workloads and simplified SGD.therefore in practical heterogeneous setting they present a limited theoretical support. also paper claims this method is useful for weak devices. but, runtime mostly supports 4 x A100's , so is it actually affordable for resource constrained clients?

Questions for authors
  • the proposed coordination depends on the past global update, however in theorem 5.1 paper assumes full client participation and equal local workloads. so when some client training got disconnected in middle and rejoins or while performing different local training steps, at that time how does FedCSO - LoRA is performing? can authors show the empirically that this method is advantageous for these heterogeneous conditions also? Given theoretical analysis accommodates heterogeneous rank budgets but does not establish the same convergence guarantee under unequal local workloads .

  • In section 7.1, it is mentioned that FSLora results are taken from the published work. however, in table 5 the paired comparsions have same random seed . so whether the FSlora accuracy is reproduced with these seed values or published results have been used? please clarify how exaclty the paired statistical tests have been done?

9 comments

We thank the reviewer for the helpful comments. Below, we explain the role of gradient energy and coordination, provide additional evidence on training with heterogeneous clients and limited GPU memory, and clarify how we reproduced FSLoRA and conducted the paired tests.

W1a: Gradient energy in practical local training

Comment: The paper selects directions that capture maximum estimated gradient energy. Proposition 4.1 showes this choice is optimal for energy.equation 31 links energy to loss decrease, but only for a single exact , uncorrected gradient step. Actual method uses a one-batch gradient estimate, proability corrected sample subspaces, and many local steps under non IID data, so this guarantee does not carry over.

Response:

We thank the reviewer for this comment. We use these three choices to keep local training efficient. First, we estimate the gradient from one mini-batch because using all local training data each time would be costly. Second, we randomly sample directions so that those outside the preferred subspace can also be updated. Probability correction ensures that the sampled update matches the full local gradient on average at the start of each round. Third, we take multiple local steps in the same subspace to reduce communication and avoid recomputing the subspace at every step.

We agree that Eq. (31) alone does not cover this complete training procedure. Below, we examine whether the mini-batch estimate provides reliable gradient information, whether retaining more gradient energy remains beneficial after probability correction, and whether the selected subspace remains useful over multiple local steps.

1). The mini-batch gradient closely aligns with the full local gradient.

At the same round-start checkpoint, we compare each client's gradient from the 16-example basis-construction batch with the gradient computed over its full local training set. As shown in Table A1, the median cosine similarity across clients ranges from 0.898 to 0.986 over the seven GLUE tasks. These results show that, at the evaluated checkpoints, the mini-batch estimate captures a direction close to that of the full local gradient, supporting its use for subspace selection at a lower computational cost.

Table A1. Cosine similarity between mini-batch and full local gradients.

TaskMRPCCoLARTESST-2QNLIMNLI-mQQP
Median cosine similarity0.9660.9270.9860.8980.9650.9260.962

2). Retaining more gradient energy improves the expected one-step descent bound after probability correction.

Probability correction makes the sampled gradient unbiased, but its variance still depends on the preferred subspace. Directions sampled less often receive larger correction weights. Keeping more gradient energy in the more frequently sampled directions reduces this variance and improves the expected descent lower bound.

Specifically, at the round-start model xx, let g=∇f(x)g=\nabla f(x) be the exact local factor gradient. Let EinE_{\mathrm{in}} and EoutE_{\mathrm{out}} denote its squared norms inside and outside the preferred subspace, respectively. Their inclusion probabilities satisfy πhi>πlo>0\pi_{\mathrm{hi}}>\pi_{\mathrm{lo}}>0. For the probability-corrected gradient g~\widetilde{g}, Eqs. (43)–(44) give

E[g~]=g,\mathbb{E}[\widetilde{g}]=g,

E∥g~∥2=Einπhi+Eoutπlo,\mathbb{E}\lVert\widetilde{g}\rVert^2=\frac{E_{\mathrm{in}}}{\pi_{\mathrm{hi}}}+\frac{E_{\mathrm{out}}}{\pi_{\mathrm{lo}}},

where the expectation is over direction sampling, with the gradient and preferred subspace held fixed.

Applying LL-smoothness to the corrected step yields

E[f(x)−f(x−ηg~)]≥η(Ein+Eout)−Lη22(Einπhi+Eoutπlo).\mathbb{E}[f(x)-f(x-\eta\widetilde{g})]\geq\eta(E_{\mathrm{in}}+E_{\mathrm{out}})-\frac{L\eta^2}{2}\left(\frac{E_{\mathrm{in}}}{\pi_{\mathrm{hi}}}+\frac{E_{\mathrm{out}}}{\pi_{\mathrm{lo}}}\right).

Writing Etot=Ein+EoutE_{\mathrm{tot}}=E_{\mathrm{in}}+E_{\mathrm{out}}, the lower bound becomes

Bcorr=(η−Lη22πlo)Etot+Lη22(1πlo−1πhi)Ein.B_{\mathrm{corr}}=\left(\eta-\frac{L\eta^2}{2\pi_{\mathrm{lo}}}\right)E_{\mathrm{tot}}+\frac{L\eta^2}{2}\left(\frac{1}{\pi_{\mathrm{lo}}}-\frac{1}{\pi_{\mathrm{hi}}}\right)E_{\mathrm{in}}.

For the same gradient, step size, and inclusion probabilities, only EinE_{\mathrm{in}} varies across candidate bases. Its coefficient is positive, so retaining more gradient energy raises the expected one-step local descent lower bound even after probability correction. This provides a direct reason to use gradient energy when selecting the preferred subspace under the corrected sampling rule. This statement concerns a fixed exact local gradient; the effects of mini-batch estimation and multi-step training are examined in the accompanying experiments.

3). The round-start basis continues to retain high gradient energy over multiple local steps.

To check whether the selected directions become less useful as local training proceeds, we keep the round-start basis fixed and measure how much gradient energy it retains at local steps 0, 1, 5, 10, and 50.

As shown in Table A2, median retention decreases from 94.87% at step 0 to 91.57% at step 50, a decrease of 3.30 percentage points. Thus, although the gradient changes during local training, the initial basis continues to capture substantial gradient energy over the evaluated trajectories.

Table A2. Gradient energy retained by the fixed round-start basis during local training.

Local step0151050
Median gradient energy retained94.87%94.79%94.13%93.64%91.57%

We also compare once-per-round refresh with per-step re-estimation. As shown in Table A3, the final GLUE accuracies are close across all seven tasks. The task mean is 85.6% with once-per-round refresh and 85.8% with per-step re-estimation, calculated from the displayed scores.

Table A3. Final GLUE accuracy by basis refresh frequency.

GLUE taskOnce-per-round refreshPer-step re-estimation
MRPC89.3%89.5%
CoLA83.8%83.9%
RTE75.1%75.5%
SST-294.1%94.3%
QNLI89.7%89.7%
MNLI-m80.9%81.0%
QQP86.3%86.4%
Task mean85.6%85.8%

In a separate timing experiment with 10 local steps per round, per-step re-estimation took 2.72–2.89 times as long per round as once-per-round refresh. These results support reusing the basis within a round: it retains substantial gradient energy and achieves accuracy close to that of per-step re-estimation, while avoiding the cost of repeatedly constructing the subspace.

We will add these analyses and experiments to the revised manuscript and clarify that Eq. (31) provides a single-step explanation, while the additional results examine the practical choices used in our training procedure.

W1b: Local gradient energy and global coordination

Comment: table 8 also shows that the cordinated basis retains less energy than the local basis bit achives higher accuracy. .so, why is gradient energy is the right proxy for the optimization quality, especially under non-IID client data.

Response:

We thank the reviewer for this valuable comment. Gradient energy measures how much local gradient information a subspace retains under a client's rank budget. We use it to select informative directions for local training. Under non-IID data, however, a direction that is useful for one client may not be equally useful for the global objective. Retaining more local gradient energy therefore does not necessarily lead to higher global accuracy. This is why we introduce coordination: it allows a small loss of local gradient energy to improve agreement with a shared update direction, while an explicit energy constraint preserves most of the local information.

The result in original manuscript Table 8 reflects this trade-off. Below, we explain how the energy constraint and coordination work together, and examine their effects on the actual updates after local training.

1). Coordination preserves most local information.

We first select the subspace that retains the most estimated local gradient energy. We then use the previous global update as a shared reference to coordinate the selected directions. This adjustment must retain at least 95% of the energy captured by the local-only basis, so improving agreement with the reference cannot come at an unrestricted loss of local information.

As shown in Table A4, coordination reduces the retained energy from 68.1% to 65.4%, preserving approximately 96.0% of the energy captured by the local-only basis. Meanwhile, mean GLUE accuracy increases from 84.7% to 85.6%. Both energy measurements refer to the preferred basis before exploration.

Table A4. Pre-exploration gradient energy and mean GLUE accuracy from Table 8 of the original manuscript.

MetricLocal-only basisCoordinated basis
Gradient energy retained68.1%65.4%
Mean GLUE accuracy84.7%85.6%

These results show that preserving the maximum local energy is not necessary for the best global accuracy. A small reduction can be worthwhile when the selected directions better support the shared training objective. The energy constraint keeps this adjustment limited and ensures that the coordinated basis still retains most of the local gradient information.

2). Global descent depends on update alignment.

For an LL-smooth global objective JJ, let g=∇J(X)g=\nabla J(X). From the same checkpoint XX, write the applied server steps as X−ηsd0X-\eta_s d_0 without coordination and X−ηsdcX-\eta_s d_c with coordination. Here, d0d_0 and dcd_c include server filtering. For either direction dd, smoothness gives

J(X−ηsd)≤J(X)−ηs⟨g,d⟩+Lηs22∥d∥2.J(X-\eta_s d)\leq J(X)-\eta_s\langle g,d\rangle+\frac{L\eta_s^2}{2}\lVert d\rVert^2.

Thus, the coordinated step has a smaller smoothness upper bound whenever

⟨g,dc−d0⟩>Lηs2(∥dc∥2−∥d0∥2).\langle g,d_c-d_0\rangle>\frac{L\eta_s}{2}\left(\lVert d_c\rVert^2-\lVert d_0\rVert^2\right).

This condition explains why a small loss of local gradient energy can be compatible with better global progress: the bound depends on the aggregate update's inner product with the global gradient and its squared norm. Maximizing each client's retained energy alone does not determine these quantities.

We also measure the norm of the applied server update after filtering. Across the tested GLUE tasks, the update with coordination has 99.07%–99.78% of the norm of the update without coordination. The quadratic term is therefore smaller in these comparisons. This measures the effect on update magnitude; the direction of the client aggregate is examined separately below.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely thank all three reviewers for their constructive feedback, which helped us improve the manuscript. We were encouraged that the reviewers saw value in learning combinations of adapter directions within the same rank budget (H2c7, syQ5) and found our approach to coordinating locally selected directions across clients well motivated (H2c7, nwnA). Reviewer syQ5 also highlighted the gains on GLUE and commonsense reasoning tasks and the supporting ablation results.

We have revised the manuscript in response to the reviewers’ suggestions. The main changes are summarized below.

1. Clarifications on the method, theory, and baseline comparisons

  • Subspace selection and exploration. We explain the purpose of exploration and inverse-probability correction (Appendix C.3) and specify when clients select the coordination coefficient (Appendix B.6.2).
  • Scope of the analysis. We clarify which assumptions of Theorem 5.1 hold in the main experiments and identify the differences in server optimization, batch sampling, and filter strength (Appendices B.1 and B.4.4).
  • Baseline comparisons. We clarify that the paired tests use our FSLoRA reruns with three matched seeds (Appendix B.3, Tables 5–8). We also distinguish our subspace selection from FedRot-LoRA’s factor alignment and explain how rank accumulation and adapter reinitialization affect the FLoRA comparison (Appendix A.4).

2. Additional experiments and measurements

  • Basis reliability. We evaluated the selected bases on independent batches, varied the estimation batch size and refresh frequency, and measured gradient energy retention over 50 local steps (Appendix B.4.5, Tables 17–20).
  • Coordination and coverage filtering. We measured conflicts between client updates and their aggregate alignment with the current estimated descent direction (Appendix B.4.2, Table 12). We also examined update noise by coverage, compared filtering with norm-matched uniform shrinkage, and tested the filter-strength schedule (Appendix B.4.4, Tables 14–16).
  • Heterogeneous training. We tested the optimizer and batch-sampling choices separately and compared FedCSO-LoRA with FSLoRA under partial participation and unequal local step counts (Appendix B.7, Tables 26–27).
  • GPU memory and runtime. We report peak client GPU memory, runtime by stage, and the memory and runtime costs of offloading. We also verified rank-8 client training on a single 11 GB consumer GPU (Appendix B.6.2, Tables 24–25).

3. Figures and notation

In Figure 1, we labeled the matrix dimensions and clarified what the alignment experiment shows. In Figure 2, we redrew Step 2 to distinguish preferred basis construction from subspace sampling and labeled the factors trained in Step 3. We also simplified the projection notation in Eq. (11).

Thank you again for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.