- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies federated LoRA, authors not only consider the misalignment caused by the hidden rotational gauge (obstacles 1 and 2), but also consider the misalignment of the update . Authors firstly show the existence of such misalignment of in Section 3.1, then propose multiple steps of operations to align the aggregation, addressing the stated 3 obstacles. Experimental results confirm the effectiveness of FLORETS.
Critical strengths
- The early sections of the paper are super clean and easy to understand. All 3 obstacles seem to be reasonable, section 3.1 is helpful for the readers to understand the subspace mismatch of . Figure 1 is super detailed to help understand the algorithm, while it may improve the paper further if you can move some equations/notations out of the figure, place them in the figure caption instead so the figure itself is not too complex.
- Authors explicitly analyze the cost of FLORETS, the cost seems to be small, and in the experiments, the performance improvement is clear.
Critical weaknesses
- The experimental results are based on 1 random seed only, which greatly hurts the reliability and strengths of the experimental results. Authors should consider present at least 3 random seeds (with resulted standard deviation across seeds).
- Section 3.2 is not clear enough and difficult to read and understand. I think authors have taken some actions to make it organized better by separating the method into 5 steps, but currently the draft only describes these steps and equations, without detailed explanation to motivate why we should do these. This could be very helpful if you can match these 5 steps with the 3 obstacles when you explain them. Also, some notations are complex, you have 2 layers of subscripts.
- Current ablation results are limited. Considering FLORETS has multiple steps of operation, I expect to see more ablation results on removing different parts.
Questions for authors
- Can you run experiments on more random seeds? Currently the results are based on one random seed, this makes your results very weak. Btw, mentioning the explicit seed (42) seems unnecessary.
- Regarding obstacle 3, can you provide more detailed explanations? I can understand obstacle 1 and 2, it is a misalignment caused by the random rotational matrix . I guess only optimizing this R to align factors (as in FedRot-LoRA) does not ensure the aligned factors are fully identical, i.e., it cannot fully address the cross-term noise (obstacle 1), it only reduces the noise. Is this the root cause of obstacle 3?
- I feel obstacle 3 is not only related to LoRA, with full parameter federated learning, you may also have "subspace misalignment" among clients, typically we simply view this as non-iid updates. Can you provide a figure similar to figure 2 and 3, but with full parameter training? With full parameter training, will and also be almost black?
- Correct me if I am wrong: heterogenous data distributions yield different client gradient directions (different ), your algorithm wants them to be aligned, that means in your operations, you may lose some local update information? like in step 5, you only keep the top for and , others are truncated.
I am happy to increase my score depending on authors' response.
8 comments
Q2 & Q3: Clarification on Obstacle 3
Thank you for these questions. As the reviewer notes, FedRot-LoRA leaves the cross-term noise of Obstacle 1, since it still averages the two factors separately after aligning them by rotation (Obstacle 2). This residual is not the cause of Obstacle 3. Obstacles 1–2 concern the LoRA factors, whereas Obstacle 3 concerns the updates themselves: under non-IID data, each client's update spans a different subspace, so the updates must be expressed in a shared basis before they can be merged coordinate by coordinate. Rotating a client's factors leaves its update unchanged, so no factor alignment can resolve Obstacle 3. Fig. 2 measures this directly: overlap is computed on the updates, so it is unaffected by any rotation, and it is near zero between clients.
On Q3, we agree that Obstacle 3 reflects non-IID heterogeneity and is not specific to LoRA; full-parameter updates would equally benefit from a shared basis. For LLMs, however, full-parameter federated training is impractical and inefficient, since exchanging full updates costs about 1.3 GB per client per round for Llama-3.2-3B against 18.3 MB for FLORETS (Table 6), which is why federated fine-tuning relies on LoRA. FLORETS therefore builds the shared basis directly from the low-rank factors. We will update Sec. 2.2 to explicitly state that Obstacles 1–2 concern the factors and Obstacle 3 the updates.
We hope this clarifies the distinction, and would be glad to discuss further if anything remains unclear.
Q4: Does alignment discard local information?
We are happy to clarify this point, as it is central to the method.
Alignment does not discard local information. It does not push client updates toward common directions; it only rewrites each client's update in a shared basis. This is an exact identity (Eq. 3), and Proposition 1 shows that the distributed computation reproduces it exactly. The shared basis spans all clients' subspaces, so every update is fully contained in it (Fig. 3, App. D.3), and client-specific content is kept in each client's own coordinates.
Information is reduced only at two later steps, both by design.
Consensus merge (Step 4) returns exactly the federated average wherever clients agree (Remark 1), and removes only contributions that oppose the majority at contested coordinates, which would otherwise offset the majority under weighted averaging.
Rank truncation (Step 5) keeps the directions above the noise level estimated by ScreeNOT. Keeping all directions instead lowers accuracy by 2.71 points (Table 3). To quantify what rank truncation retains, we measure the fraction of each client's update energy kept by the deployed adapter at the selected rank (Dolly / Llama-3.2-1B, α=0.1; query and value projections at the first, middle and last layers). By the final round, FLORETS retains 98–99% of each client's energy on average, and at least 97% for every client. At every layer and round, it is higher than when the summed aggregate is truncated to the same rank, as in FLoRIST and SpecTraL. We will add this analysis to the appendix.
We hope this clarifies the question, and would be glad to discuss further if anything remains unclear.
W2: Clarity of Section 3.2
Thank you for the suggestion. We will be happy to incorporate the following motivations for the steps of FLORETS into the revised manuscript.
Steps 1–3: Distributed cross-client alignment in low-rank space. A merge that compares clients coordinate by coordinate needs three things: each client's update must remain separate and exact, all updates must be written in one shared basis, and this must be done efficiently every round, without forming full-size matrices and without placing the entire computation on a single server. The usual vertical stack fails the first requirement, because it returns only the summed update and can no longer tell which client contributed what. Block-diagonal stacking of the A factors meets it: it gives the exact concatenation of all client updates, free of cross-terms (Obstacle 1) and of any reparameterization of the factors (Obstacle 2), and it accommodates different client ranks (Step 2). The second requirement arises because client updates lie in different subspaces and their low-rank factors use arbitrary, client-specific coordinates, so these coordinates cannot be compared directly; expressing every update in one shared basis makes each coordinate refer to the same direction for every client (Step 3, Obstacle 3). The third requirement motivates the local QR: each client orthogonalizes its own factors, so this work is distributed and parallelized across the clients rather than carried by a single server, which then performs only one small cross-client QR and one small SVD (Step 1).
Steps 4–5: Server-side construction of the global LoRA adapter for efficient broadcasting. The aligned updates must then be combined into a single adapter that is both informed by the alignment and compact enough to broadcast every round. A weighted sum is invariant to the choice of basis, so it would return the same aggregate with or without alignment (Eq. 4); alignment can affect the result only through a coordinate-wise rule, which the consensus merge provides (Step 4). The merged update has rank equal to the sum of the client ranks, so a compact broadcast requires truncation, and the retained rank is chosen from the data rather than fixed in advance (Step 5).
Notation. We will remove nested subscripts, writing, for example, RA,k and RB,k for the client QR factors. As the reviewer suggests, we will also move the equations inside Figure 1 to its caption.
Could the reviewer let us know whether these changes address the concern, or whether anything else would help? We will then update Section 3.2 accordingly in the revised manuscript.