- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies federated LoRA for heterogeneous devices: different devices can support different LoRA ranks, and this paper optimizes a projection matrix Q_i to adjust the dimension of B_i and A_i (forming BQ and Q^T*A). To optimize the matrix Q_i, this paper mainly considers 2 directions: (1) Q_i should maintain enough "full gradient" energy (Eq. 4), and (2) the projection Q_i should make sure the client subspaces are aligned (Eq. 7-9). Their solution is motivated well, with theoretical and experimental results to support the effectiveness of the algorithm.
Critical strengths
The most of the paper is clear to understand and authors unfold their main algorithm (section 4.1 and 4.2) in a good structure, the problem is meaningful for the topic of federated LoRA. I agree on authors' idea to optimize Q for both maintaining gradient energy and subspace alignment, from my knowledge of this field, this is a clear and well motivated solution, so the good experimental results are reasonable.
Critical weaknesses
- Figure 1 and 2 should be improved. I don't get the point of Figure 1 (c). For figure 1 (a), it would be helpful to clarify the dimension of B, Q in the figure. For Figure 2, step 2 seems conflict with the algorithm you described. In Eq. 9, your optimization is one-step, while in figure 2 step 2, you separate this into two optimization steps (local-coordinated). Also in Figure 2, place specify what these rectangles are, B, A, Q?
- The algorithm seems to be expensive as it requires one round of gradient computation in the large rank R space (contradict with the assumption that the device cannot afford the training of rank R), and also the coefficient c is decided by a greedy search (is this search per-round or one-time for the whole training?).
- Vague motivation for the exploration (line 273 to 276). If you solve Eq. 9, you get the optimal solution in terms of your objective, what is the aim of this exploration and sampling?
- Authors defined some unnecessary and redundant notations that actually stop human readers to understand the paper, like P=QQ^T.
Questions for authors
- Please explain the motivation for using the exploration (line 273 to 276).
- Can authors provide clearer explanation for the cost of FedCSO-LoRA? Theoretical complexity or more detailed experimental metrics other than GPU hour?
- One recent work FedRot-LoRA also optimizes an orthonormal basis Q_i, what's the difference between their optimization and your algorithm? I feel they mostly focus on the subspace alignment (like your section 4.2), is FedRot-LoRA a complementary method for your setting or you may want to compare it in the experiments?
I think the paper's idea is meaningful and good, while many issues should be addressed before the paper reaches the acceptance bar.
7 comments
W2: Full-rank gradient estimation and search frequency
Comment: The algorithm seems to be expensive as it requires one round of gradient computation in the large rank R space (contradict with the assumption that the device cannot afford the training of rank R), and also the coefficient c is decided by a greedy search (is this search per-round or one-time for the whole training?).
Response:
We estimate gradients and select at round start, then keep the resulting subspace fixed throughout local training. Rank-wise block estimation and layer-wise offloading reduce GPU memory requirements, while the search for uses precomputed matrices without additional backbone forward or backward passes. Our measurements show modest overall runtime overhead.
1). Rank- gradient estimation has modest overhead.
We use one mini-batch to estimate the rank- factor gradients for subspace selection. This computation is performed at round start and is not repeated at every local step. In the manuscript’s benchmark configuration, the full method uses 11.3 GPU hours on GLUE and 45.8 GPU hours on commonsense reasoning, compared with 10.9 and 44.3 GPU hours for FSLoRA. These increases of approximately 3.7% and 3.4% include the cost of gradient estimation and subspace construction.
To accommodate clients with limited GPU memory, we provide two strategies.
(a) Rank-wise block estimation. Computing the gradients over all directions does not require all of them to be active simultaneously on the GPU. As described in Appendix A.3, each backward pass computes gradients for at most active directions, with the remaining directions folded into the frozen weights. All blocks use the same round-start model and estimation batch. Completed gradient blocks are moved to host memory, avoiding the need to allocate the complete rank- factor gradients on the GPU.
(b) Layer-wise backbone offloading. For tighter GPU memory budgets, we also evaluated layer-wise BF16 offloading. In the one-local-step benchmark in Table C1, a gradient-estimation microbatch size of 4 reduces peak allocated GPU memory by 64.24%, with a 12.60% increase in total runtime. A microbatch size of 2 further reduces memory by 76.67%, at a higher runtime cost. Here, microbatch size denotes the number of examples processed together during gradient estimation.
Table C1. Peak PyTorch allocated GPU memory and runtime for a short run with one local step on the LLaMA-3.2-3B commonsense reasoning client.
| Configuration | Gradient-estimation microbatch size | Gradient estimation time | Total runtime with one local step | Peak allocated GPU memory |
|---|---|---|---|---|
| GPU-resident BF16 reference | 1 | 3.07 s | 11.98 s | 7,435.64 MiB |
| Layer-wise BF16 offloading | 2 | 4.83 s | 15.40 s | 1,735.02 MiB |
| Layer-wise BF16 offloading | 4 | 2.71 s | 13.49 s | 2,659.16 MiB |
In a separate FP32 GLUE experiment, layer-wise offloading achieved the same accuracy as the GPU-resident configuration at the reported precision.
2). The coordination search has low overhead.
The search for uses precomputed matrices and bounded expansion and bisection when needed. Testing candidate values requires no additional backbone forward or backward passes. In our MRPC benchmark with 20 participating clients, the search across all clients and layers took 2.52 s per round, about 3.1% of the median round time of 81.25 s.
In GLUE, each participating client runs this search once per layer at the start of each round from round 2 onward. The first round uses local gradient selection because no historical reference is available. The resulting training subspace remains fixed for all 50 local SGD steps.
W3 / Q1: Motivation for exploration
Comment (W3): Vague motivation for the exploration (line 273 to 276). If you solve Eq. 9, you get the optimal solution in terms of your objective, what is the aim of this exploration and sampling?
Comment (Q1): Please explain the motivation for using the exploration (line 273 to 276).
Response:
Equation (9) of the original manuscript uses the estimated local gradient and the global reference to select a preferred subspace within the client’s rank budget. Even when this selection is optimal, directions outside the subspace may still contain gradient information.
1). Exploration allows updates outside the preferred subspace.
Exploration gives these remaining directions a chance to contribute, while each sampled subspace still contains only directions. We then weight each selected direction by the inverse of its inclusion probability. As shown in Eq. (43) of the original manuscript, the corrected estimate recovers the full local factor gradient in expectation over direction sampling at round start. Without exploration, gradient components outside the preferred subspace would be excluded and could not be recovered by this correction.
2). Energy-guided selection controls sampling variance.
Correction preserves the mean gradient, but the sampling probabilities still determine how often each direction contributes and how much it is weighted. Preferred directions are selected more often and receive smaller correction weights. For a fixed gradient and fixed inclusion probabilities, Eq. (44) of the original manuscript shows that placing more gradient energy in these frequently sampled directions reduces the variance of the corrected estimate. Thus, exploration and correction preserve the full gradient in expectation, while energy-guided selection helps reduce the expected squared error for the gradient used to construct the basis.
W4: Notation
Comment: Authors defined some unnecessary and redundant notations that actually stop human readers to understand the paper, like P=QQ^T.
Response:
We will simplify the notation in the main text by replacing with , avoiding a separate symbol for a quantity already determined by . For example, Eq. (11) of the original manuscript will read . We will also use consistent symbols for the factors and bases across the text and figures.
Q2: Memory and runtime costs
Comment: Can authors provide clearer explanation for the cost of FedCSO-LoRA? Theoretical complexity or more detailed experimental metrics other than GPU hour?
Response:
To clarify the cost beyond GPU hours, we compare FSLoRA and FedCSO-LoRA stage by stage on MRPC and report peak GPU memory for FedCSO-LoRA. We also provide a separate experiment showing the memory and runtime trade-off of offloading.
1). Complete-round runtime and memory.
Our MRPC benchmark uses 20 participating clients per round and 50 local training steps per client. As shown in Table C2, the median round time is 81.25 s for FedCSO-LoRA and approximately 75.55 s for FSLoRA, an increase of about 5.70 s. Local training accounts for most of the runtime in both methods, taking 76.73 s for FedCSO-LoRA and 75.55 s for FSLoRA. Among the additional stages in FedCSO-LoRA, coordination search takes 2.52 s, the largest stage cost outside local training. FedCSO-LoRA's peak PyTorch allocated GPU memory is 2,368.62 MiB.
Table C2. Median stage and complete-round runtimes for FSLoRA and FedCSO-LoRA on MRPC.
| Stage | FSLoRA (s) | FedCSO-LoRA (s) |
|---|---|---|
| Gradient estimation | — | 1.40 |
| Basis construction | — | 0.34 |
| Coordination search | — | 2.52 |
| Exploration | — | 0.18 |
| Local training | 75.55 | 76.73 |
| Server update | 0.002 | 0.075 |
| Complete round | 75.55 | 81.25 |
Stage medians and complete-round medians are computed separately. A dash indicates that the stage is absent in FSLoRA.
2). Memory and runtime with offloading.
For clients with tighter GPU memory budgets, we evaluate layer-wise BF16 offloading on LLaMA-3.2-3B for commonsense reasoning. This separate benchmark includes gradient estimation and one local training step.
As reported in Table C1 of our response to W2, the offloaded configuration with a gradient-estimation microbatch size of 4 reduces peak allocated GPU memory from 7,435.64 MiB to 2,659.16 MiB, while runtime increases from 11.98 s to 13.49 s. This corresponds to 64.24% less GPU memory and 12.60% longer runtime in this benchmark. A microbatch size of 2 further reduces the peak to 1,735.02 MiB, with a runtime of 15.40 s, providing another option when memory is more limited.
We will add these measurements and their configurations to the revised manuscript.
Thank you for the detailed feedback. We have revised Figures 1 and 2, clarified why we use exploration after solving Eq. (9), and added measurements of peak GPU memory and runtime.
W1: Figures 1 and 2
Comment: Figure 1 and 2 should be improved. I don't get the point of Figure 1 (c). For figure 1 (a), it would be helpful to clarify the dimension of B, Q in the figure. For Figure 2, step 2 seems conflict with the algorithm you described. In Eq. 9, your optimization is one-step, while in figure 2 step 2, you separate this into two optimization steps (local-coordinated). Also in Figure 2, place specify what these rectangles are, B, A, Q?
Response:
We have revised Figures 1 and 2 and their captions as follows.
1). Figure 1 (a) now labels B, Qi, and BQi with dimensions m×R, R×ki, and m×ki, respectively. Both rows illustrate R=6 and ki=2.
2). The revised caption of Figure 1 (c) defines the horizontal axis as the cosine similarity between the projected local gradient and the previous global update reference. The vertical axis shows accuracy changes in percentage points. On both tasks, accuracy first improves and then declines as alignment increases. This trend motivates limiting the loss of local gradient information during coordination.
3). Equation (9) constructs each candidate basis using local gradient information and the global reference together. The local-only case c=0 provides the energy baseline for the acceptance conditions in Eq. (10). We have therefore redrawn Step 2 to show both inputs leading to preferred basis construction, followed by subspace sampling. This replaces the potentially misleading “Local → Coordinated” sequence.
4). The revised caption of Figure 2 identifies the vertical and horizontal rectangles as Bt and At in the global adapter, and Bˉi and Aˉi in Step 3. In Step 2, Qi⋆ denotes the preferred basis and Qi denotes the sampled training basis.