acceptodds
Under review as a conference paper at ICLR 2027

FLORETS: Weaving Cross-Client Low-Rank Adapters into an Aligned Subspace for Federated LLMs

Abstract

Federated fine-tuning of Large Language Models with low-rank adapters faces a fundamental challenge: updates from client adapters trained on non-IID data lie in misaligned representational bases. Conventional schemes aggregate these adapters via weighted averaging without aligning their bases, leading to destructive parameter interference. We propose a novel federated framework, FLORETS (Federated LOw-Rank adaptation with Election and Truncation for Subspace alignment) for aligning client updates within a shared basis entirely inside the low-rank adapter subspace, without ever reconstructing full-dimensional matrices. FLORETS uses a block-diagonal stacking mechanism %to handle heterogeneous client ranks to uncover the shared basis through a distributed Householder QR decomposition that parallelizes %orthogonalization construction of the shared subspace amongst multiple clients. In contrast to simple weighted averaging, FLORETS aggregates the aligned updates using a coordinate-wise, consensus-driven sign election followed by rank selection. Extensive evaluations across Llama-3.2 (1B and 3B) on Dolly and GSM8K under severe Dirichlet non-IID settings demonstrate that FLORETS consistently outperforms existing federated LoRA baselines in downstream accuracy and computational overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated Learning, LoRA, Parameter-Efficient Fine-Tuning, Subspace Alignment, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RbitOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies federated LoRA, authors not only consider the misalignment caused by the hidden rotational gauge (obstacles 1 and 2), but also consider the misalignment of the update ΔWi\Delta W_i. Authors firstly show the existence of such misalignment of ΔWi\Delta W_i in Section 3.1, then propose multiple steps of operations to align the aggregation, addressing the stated 3 obstacles. Experimental results confirm the effectiveness of FLORETS.

Critical strengths
  1. The early sections of the paper are super clean and easy to understand. All 3 obstacles seem to be reasonable, section 3.1 is helpful for the readers to understand the subspace mismatch of ΔWi\Delta W_i. Figure 1 is super detailed to help understand the algorithm, while it may improve the paper further if you can move some equations/notations out of the figure, place them in the figure caption instead so the figure itself is not too complex.
  2. Authors explicitly analyze the cost of FLORETS, the cost seems to be small, and in the experiments, the performance improvement is clear.
Critical weaknesses
  1. The experimental results are based on 1 random seed only, which greatly hurts the reliability and strengths of the experimental results. Authors should consider present at least 3 random seeds (with resulted standard deviation across seeds).
  2. Section 3.2 is not clear enough and difficult to read and understand. I think authors have taken some actions to make it organized better by separating the method into 5 steps, but currently the draft only describes these steps and equations, without detailed explanation to motivate why we should do these. This could be very helpful if you can match these 5 steps with the 3 obstacles when you explain them. Also, some notations are complex, RAKR_{A_K} you have 2 layers of subscripts.
  3. Current ablation results are limited. Considering FLORETS has multiple steps of operation, I expect to see more ablation results on removing different parts.
Questions for authors
  1. Can you run experiments on more random seeds? Currently the results are based on one random seed, this makes your results very weak. Btw, mentioning the explicit seed (42) seems unnecessary.
  2. Regarding obstacle 3, can you provide more detailed explanations? I can understand obstacle 1 and 2, it is a misalignment caused by the random rotational matrix RR. I guess only optimizing this R to align factors (as in FedRot-LoRA) does not ensure the aligned factors are fully identical, i.e., it cannot fully address the cross-term noise (obstacle 1), it only reduces the noise. Is this the root cause of obstacle 3?
  3. I feel obstacle 3 is not only related to LoRA, with full parameter federated learning, you may also have "subspace misalignment" among clients, typically we simply view this as non-iid updates. Can you provide a figure similar to figure 2 and 3, but with full parameter training? With full parameter training, will ΔWa\Delta W_a and ΔWb\Delta W_b also be almost black?
  4. Correct me if I am wrong: heterogenous data distributions yield different client gradient directions (different ΔWi\Delta W_i), your algorithm wants them to be aligned, that means in your operations, you may lose some local update information? like in step 5, you only keep the top r∗r^* for BgB_g and AgA_g, others are truncated.

I am happy to increase my score depending on authors' response.

8 comments
AuthorsOpenReview ↗

W2: Clarity of Section 3.2

Thank you for the suggestion. We will be happy to incorporate the following motivations for the steps of FLORETS into the revised manuscript.

Steps 1–3: Distributed cross-client alignment in low-rank space. A merge that compares clients coordinate by coordinate needs three things: each client's update must remain separate and exact, all updates must be written in one shared basis, and this must be done efficiently every round, without forming full-size matrices and without placing the entire computation on a single server. The usual vertical stack fails the first requirement, because it returns only the summed update and can no longer tell which client contributed what. Block-diagonal stacking of the A factors meets it: it gives the exact concatenation of all client updates, free of cross-terms (Obstacle 1) and of any reparameterization of the factors (Obstacle 2), and it accommodates different client ranks (Step 2). The second requirement arises because client updates lie in different subspaces and their low-rank factors use arbitrary, client-specific coordinates, so these coordinates cannot be compared directly; expressing every update in one shared basis makes each coordinate refer to the same direction for every client (Step 3, Obstacle 3). The third requirement motivates the local QR: each client orthogonalizes its own factors, so this work is distributed and parallelized across the clients rather than carried by a single server, which then performs only one small cross-client QR and one small SVD (Step 1).

Steps 4–5: Server-side construction of the global LoRA adapter for efficient broadcasting. The aligned updates must then be combined into a single adapter that is both informed by the alignment and compact enough to broadcast every round. A weighted sum is invariant to the choice of basis, so it would return the same aggregate with or without alignment (Eq. 4); alignment can affect the result only through a coordinate-wise rule, which the consensus merge provides (Step 4). The merged update has rank equal to the sum of the client ranks, so a compact broadcast requires truncation, and the retained rank is chosen from the data rather than fixed in advance (Step 5).

Notation. We will remove nested subscripts, writing, for example, RA,kR_{A,k} and RB,kR_{B,k} for the client QR factors. As the reviewer suggests, we will also move the equations inside Figure 1 to its caption.

Could the reviewer let us know whether these changes address the concern, or whether anything else would help? We will then update Section 3.2 accordingly in the revised manuscript.

AuthorsOpenReview ↗

Q2 & Q3: Clarification on Obstacle 3

Thank you for these questions. As the reviewer notes, FedRot-LoRA leaves the cross-term noise of Obstacle 1, since it still averages the two factors separately after aligning them by rotation (Obstacle 2). This residual is not the cause of Obstacle 3. Obstacles 1–2 concern the LoRA factors, whereas Obstacle 3 concerns the updates themselves: under non-IID data, each client's update spans a different subspace, so the updates must be expressed in a shared basis before they can be merged coordinate by coordinate. Rotating a client's factors leaves its update unchanged, so no factor alignment can resolve Obstacle 3. Fig. 2 measures this directly: overlap is computed on the updates, so it is unaffected by any rotation, and it is near zero between clients.

On Q3, we agree that Obstacle 3 reflects non-IID heterogeneity and is not specific to LoRA; full-parameter updates would equally benefit from a shared basis. For LLMs, however, full-parameter federated training is impractical and inefficient, since exchanging full updates costs about 1.3 GB per client per round for Llama-3.2-3B against 18.3 MB for FLORETS (Table 6), which is why federated fine-tuning relies on LoRA. FLORETS therefore builds the shared basis directly from the low-rank factors. We will update Sec. 2.2 to explicitly state that Obstacles 1–2 concern the factors and Obstacle 3 the updates.

We hope this clarifies the distinction, and would be glad to discuss further if anything remains unclear.

AuthorsOpenReview ↗

Q4: Does alignment discard local information?

We are happy to clarify this point, as it is central to the method.

Alignment does not discard local information. It does not push client updates toward common directions; it only rewrites each client's update in a shared basis. This is an exact identity (Eq. 3), and Proposition 1 shows that the distributed computation reproduces it exactly. The shared basis spans all clients' subspaces, so every update is fully contained in it (Fig. 3, App. D.3), and client-specific content is kept in each client's own coordinates.

Information is reduced only at two later steps, both by design.

Consensus merge (Step 4) returns exactly the federated average wherever clients agree (Remark 1), and removes only contributions that oppose the majority at contested coordinates, which would otherwise offset the majority under weighted averaging.

Rank truncation (Step 5) keeps the directions above the noise level estimated by ScreeNOT. Keeping all directions instead lowers accuracy by 2.71 points (Table 3). To quantify what rank truncation retains, we measure the fraction of each client's update energy kept by the deployed adapter at the selected rank (Dolly / Llama-3.2-1B, α=0.1; query and value projections at the first, middle and last layers). By the final round, FLORETS retains 98–99% of each client's energy on average, and at least 97% for every client. At every layer and round, it is higher than when the summed aggregate is truncated to the same rank, as in FLoRIST and SpecTraL. We will add this analysis to the appendix.

We hope this clarifies the question, and would be glad to discuss further if anything remains unclear.

Reviewer_WBEvOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes FLORETS for aggregating client LoRA updates in federated fine-tuning. Its design has four main parts:

  1. Construct a shared basis using concatenation followed by SVD.
  2. Merge coefficients using sign consensus.
  3. Rescale the merged update.
  4. Select the retained rank using ScreeNOT.

Experiments use Llama-3.2-1B and 3B on Dolly fine-tuning evaluated with MMLU, and on GSM8K, under two levels of data heterogeneity. FLORETS achieves the highest accuracy in seven of eight main settings, generally with modest gains over the strongest baseline. Component ablations provide support the usefulness of the each parts.

Critical strengths
  • FLORETS wins seven of eight main configurations against several federated LoRA baselines. Most gains are modest, and single-seed results limit confidence, but the consistency is still positive evidence for the complete recipe.
  • The paper combines existing merging techniques into a practical federated LoRA aggregation procedure, with attention to computational and communication costs. It provides motivations for the individual design choices and some empirical support through ablation studies.
Critical weaknesses
  • The paper does not establish a convincing connection between its motivating observations, design choices, and performance gains. Figure 2 shows limited overlap between client subspaces, but this does not necessarily imply harmful interference or poor aggregation. The paper does not directly demonstrate that the observed misalignment causes a performance loss that its shared-basis construction resolves. Similarly, sign-based consensus and norm correction are plausible heuristics, but their connection to improved optimization or generalization remains insufficiently supported. The ablations show that removing components can hurt performance within the proposed pipeline; they do not establish the claimed mechanism or isolate the advantage of concatenation-based basis extraction. Given the close relationship of these components to existing techniques, comparisons with closely matched shared-subspace merging methods such as KnOTS are also important. Consequently, the paper demonstrates a potentially useful recipe, but offers limited new understanding of federated LoRA aggregation or evidence establishing its substantive advance.
  • The empirical evidence is not strong enough to establish a reliable practical advantage. Although FLORETS achieves the highest accuracy in seven of eight main settings, most gains over the strongest baseline are below one percentage point, and all configurations use a single training seed. Consequently, the reliability of these improvements remains uncertain. The efficiency evidence also relies primarily on analytical cost estimates: QR does not materially reduce upload volume, and download savings depend on the selected rank and comparator. Comparisons at matched communication budgets, or measurements of total cost to reach a target accuracy, would be needed to establish the claimed practical benefit. Since empirical and efficiency gains carry much of the contribution, these limitations materially weaken the case for acceptance.
Questions for authors
  1. What makes the near-orthogonality in Figure 2 harmful? The updates already occupy the same model-parameter coordinates, and orthogonal directions need not interfere destructively when added. What evidence distinguishes harmful aggregation from useful client diversity in this figure, and connects the measured overlap to downstream performance?
  2. Can you clarify both the motivation and implementation of norm correction? Appendix C.3 says discarding opposing contributions reduces the aggregate’s norm. However, retaining contributions matching the elected sign increases or preserves each coordinate’s magnitude relative to the signed sum—for example, merging (3) and (-2) gives (3), rather than (1). Also, Equation 3 already incorporates (p_k), so Equation 5 appears to set the target to (\sum_k p_k^2|\Delta W_k|_F). Is this additional weighting intentional and used in the experiments? This matters particularly because normalization produces the largest ablation effect.
  3. In what sense does the concatenation spectrum measure “genuine inter-client agreement”? Step 5 makes this claim, but replacing any client update by its negative leaves the concatenation’s singular values unchanged. Thus, perfectly agreeing updates and perfectly opposing updates can have identical spectra. What notion of agreement is intended, and how does it justify the rank-selection rule? 4.What optimality does truncation retain after nonlinear consensus? The singular values used for ordering and ScreeNOT come from the concatenated updates before sign merging. After merging, the right factor generally no longer has orthonormal rows. Why should retaining the original leading directions give the claimed “best rank-(r^) approximation”? Does this mean only projection onto a prescribed subspace, or an optimal low-rank approximation of the merged update? 5.Does the reported server cost include full-rank consensus and normalization? Algorithm 1 constructs the aligned factors, merges them, and normalizes before truncation. Appendix C.5 instead counts factor reconstruction and consensus using only (r^) retained directions. Which order does the implementation follow? If it truncates first, how is Equation 5’s full-rank normalization denominator obtained without processing the discarded directions?
6 comments
AuthorsOpenReview ↗

Response to W1 & Q1: How the motivating observation connects to the design and to the gains

We would like to clarify that the paper does not present Fig. 2 as evidence of harmful interference. As Eq. 4 shows, a weighted sum is invariant to the choice of basis, so near-orthogonality neither harms nor helps a linear aggregate. Fig. 2 is instead used to draw two design consequences: no single client's basis can serve as the shared basis, and the vertical stack, which sums the clients, cannot keep them separate for a coordinate-wise merge.

(Additional Ablations) The following ablations test each consequence directly and further demonstrate the effectiveness of FLORETS (Dolly / Llama-3.2-1B, α=0.1, MMLU %; details in Sec. 4 of the revised manuscript and in our response to Reviewer Rbit, W3):

  • Without alignment (consensus merge on the full updates): 34.36 → 26.67.
  • Vertical stacking instead of block-diagonal stacking: 34.36 → 21.02. Both bases span the same subspace and differ only by a rotation, so this isolates the advantage of concatenation-based basis extraction.
  • Single-client basis: 34.36 → 17.06, since near-orthogonal subspaces leave most other clients' updates outside it.

All three fall below a plain linear sum (29.79), while the consensus merge in the FLORETS basis improves on it by 4.57 points (Table 3): a shared basis helps only together with a coordinate-wise merge, and only when it keeps the clients separate.

(Additional Baseline) KnOTS-TIES, the closest shared-basis merging method, reaches 25.05, so the federated merge design of FLORETS matters as well as the basis.

(Additional Analysis) FLORETS also preserves client diversity: by the final round, the broadcast adapter retains 98–99% of each client's update energy on average, and at least 97% for every client, more than when the summed aggregate is truncated to the same rank, as in FLoRIST and SpecTraL. Norm matching is addressed in our response to Q2.

We have added the ablations to Table 3 (Sec. 4) and the retention analysis to App. D.4, and rewritten Sec. 3.2 so that each step opens with the reason it is needed.

Could the reviewer confirm whether this addresses the concern?

AuthorsOpenReview ↗

Response to Q2: Motivation and implementation of norm correction

(Writing and Implementation) Thank you for bringing this to our notice: Eq. 5 in App. C.3 contains a writing typo. However, our code implementation follows the correct methodology, and all reported results remain unaffected. Norm matching applies no additional pk weighting, and its target is not a sum of client norms: it rescales the merged factor so that the merged update has the Frobenius norm of the exact federated aggregate (Eq. 4). Because the shared basis has orthonormal columns, both norms are computed in the low-rank space. The corrected text, shown at the end of this response, has been updated in the revised manuscript.

(Motivation) The motivation is to match the step size of the exact federated aggregate. The elected sign at each coordinate is the sign of the weighted sum, so every discarded contribution opposes it, and its removal increases the magnitude at that coordinate, as in the reviewer's example. The merged update is therefore larger in norm than the exact aggregate wherever clients disagree. Norm matching keeps the direction chosen by the consensus and restores the step size of the exact aggregate, with no tuned hyperparameter. Removing it changes only the scale of the update, not its direction, and costs 14.91 points (Table 3).

(Revised App. C.3)

Because the elected sign at each coordinate is the sign of the weighted sum, every excluded contribution opposes it. Excluding dissenting clients therefore increases the magnitude at each contested coordinate, and the consensus factor is larger in Frobenius norm than the exact aggregate it replaces. We restore the step size of the exact aggregate with a single global factor per adapted layer,

V⋆←ν∥ΣV⋆∥V⋆,ν=∥ΔW∗∥,(5)V^{\star} \leftarrow \frac{\nu}{\lVert \Sigma V^{\star} \rVert} V^{\star}, \qquad \nu = \lVert \Delta W^{*} \rVert, \qquad (5)

where is the Frobenius norm of the exact federated aggregate of Eq. 4. …

Could the reviewer confirm whether this resolves the question?

AuthorsOpenReview ↗

Response to Q3 & Q4: Rank selection and truncation

(Writing and Implementation) Both questions arise from a single oversight in the written description of Step 5; thank you for bringing this to our notice. However, our code implementation includes this step and follows the correct methodology, and every reported result remains unaffected. After the consensus merge and norm matching, the merged update is factorized again by a thin SVD, and rank selection and truncation operate on this factorization. The concatenation is used only to build the shared basis in which clients are aligned. We have added this step to Step 5 (Sec. 3.2), Algorithm 1 (App. C) and the complexity analysis (App. C.4–C.5), and revised Figure 1 accordingly; the revised text is shown at the end of this response.

Q3. ScreeNOT is applied to the spectrum of the merged update, computed after the sign election, so it reflects agreement directly. In the reviewer's example of two equally weighted, exactly opposing clients, every elected sign is zero and the merged update vanishes; for agreeing clients, the full energy is kept. Rank selection therefore keeps the directions of the agreed update that stand above the noise level, without a tuned threshold. Removing truncation costs 2.71 points (Table 3).

Q4. Because the shared basis has orthonormal columns, the thin SVD of the merged factor is the exact SVD of the merged update. Truncating it to its leading r* singular triplets gives the best rank-r* approximation of the merged update (Eckart–Young), not only a projection onto a prescribed subspace.

(Additional Analysis) The broadcast adapter retains 98–99% of each client's update energy by the final round, and at least 97% for every client, more than when the summed aggregate is truncated to the same rank, as in FLoRIST and SpecTraL.

(Revised Step 5, Sec. 3.2)

… chosen from the data. After the consensus merge and norm matching, we factorize the merged factor again by a thin SVD, ΣCV^⋆⊤=UMSMVM⊤\Sigma_C \widehat{V}_{\star}^{\top} = U_M S_M V_M^{\top}, and rotate the shared basis to U^UM\widehat{U} U_M; this is the exact SVD of the merged update. We select the retained rank from this spectrum by random-matrix denoising … Truncating this SVD to its leading r∗r^{\ast} singular triplets gives Bg=U^UMSMB_g = \widehat{U} U_M S_M and Ag=VM⊤A_g = V_M^{\top}, restricted to their first r∗r^{\ast} columns and rows, which is the best rank-r∗r^{\ast} approximation of the merged update. …

(Revised Algorithm 1, App. C; lines 15–16 are new, and lines 17–18 now use the merged SVD)

14: Match norm: V^⋆⊤←\widehat{V}_{\star}^{\top} \gets Eq. (5)

15: Re-factorize: ΣCV^⋆⊤=UMSMVM⊤\Sigma_C \widehat{V}_{\star}^{\top} = U_M S_M V_M^{\top}, with s1≥⋯≥srs_1 \ge \cdots \ge s_r

16: Rotate basis: U^←U^UM\widehat{U} \gets \widehat{U} U_M

17: Rank: r∗←r^{\ast} \gets ScreeNOT$(s_1, \dots, s_r)$

18: Reconstruct: Bg←U^UMSMB_g \gets \widehat{U} U_M S_M, Ag←VM⊤A_g \gets V_M^{\top}, truncated to rank r∗r^{\ast}

Could the reviewer confirm whether this resolves both questions?

Reviewer_Ti59OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a federated learning framework, FLORETS, that aims to resolve the three challenges in existing federated-LoRA finetuning literatures: cross-term noise, rotational gauge, and subspace misalignment of updates. FLORETS leverages SVD-style of aggregation but all clients share a common subspace UU with client-specific VkV_k. FLORETS further adopts consensus merge technique that only retains the entries of VkV_k that having same signs for average.

Critical strengths
  1. The paper identifies essential issue of existing federated fine-tuning literatures, and FLORETS offers a simple and effective approach to resolve these issue.
  2. The empirical results show that FLORETS is able to outperform existing baselines across different datasets and data heterogeneity settings.
Critical weaknesses
  1. It seems a previous method, FFA-LoRA, is also a method that fixes subspace and ensure exact aggregation, which resolves the same challenges as FLORETS. Yet, FFA-LoRA does not have the consensus merge component. From Table 3’s ablation result, it seems after removing consensus merge, FLORETS slightly underperforms than FFA-LoRA. In this case, could the author confirm whether the effectiveness of FLORETS comes from its step 1-3 or consensus merge(step 4)? If we apply FFA-LoRA+consensus merge, will a better result be obtained?
  2. FLORETS is algorithmically similar to KnOTS, though KnOTS is applied to model merging setting. Since model merging field and federated learning field share some similarity, could the author justify more on the difference between FLORETS and KnOTS other than field difference? I want to assess the novelty of FLORETS from author’s response.
  3. I think there are also many other papers that works on federated-LoRA aggregation that aims for either exact aggregation or subspace alignment. For example: Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA. It will be better for author to consider more comprehensive related work in the paper.
  4. Further justification for the effectiveness of consensus merge is expected from the author. It seems to me that the sign-based selection strategy introducing some nonlinearity into the aggregation scheme, but it remains unclear how it contributes to better performance. Heuristically, the sign symbols consensus group, but omitting entries with different signs may also neglect important information.
Questions for authors

Can author also test FLORETS on heterogeneous ranks and compare its performance with heterogeneous rank baselines?

6 comments
AuthorsOpenReview ↗

Response to Weakness W3: More comprehensive related work, e.g., GLoRA

Thank you for bringing GLoRA to our attention. We would like to highlight that our Appendix B (and Table 7) comprises a comprehensive list of 18 related works. GLoRA is a relevant addition, and we will include it in Appendix B. Since no public implementation is available in its preprint, we are unable to verify and replicate its results. However, we discuss the methodological differences below:

  • Geometry-only vs. energy-weighted basis: GLoRA builds its basis from client projectors, recording which directions each update spans but not how strongly it moves along them. Hence, strong and noise-level directions count equally - a limitation GLoRA explicitly lists as future work. FLORETS by design resolves this by computing an exact, energy-weighted basis that perfectly matches the centralized SVD (Proposition 1) using efficient distributed QR decomposition across the clients and server.
  • Linear vs. consensus merge: GLoRA applies linear averaging to projected coordinates, which fails to exploit the shared basis (our Eq. 4) thereby reducing it to a standard projected average. In contrast, FLORETS uses the shared basis to actively compare and merge client updates via sign consensus followed by norm correction. Our results demonstrate that such consensus merge is critical to performance (Table 3).
  • Server state and rank: GLoRA requires maintaining historical per-client bases on the server, whereas FLORETS is completely stateless. Furthermore, GLoRA's server rank is a fixed hyperparameter, whereas, FLORETS dynamically selects the optimal rank per layer using tuning-free ScreeNOT.

We hope this addresses the concerns raised.

AuthorsOpenReview ↗

Response to W2: FLORETS and KnOTS

While we credit KnOTS for utilizing a common basis for centralized model merging in our Appendix B.5, its procedure cannot be effectively applied to federated learning. In contrast, FLORETS contributes the novel mechanisms required to transform a one-shot shared-basis merge of KnOTS into a practical, repeated federated LoRA aggregator. We summarize the key differences below:

  1. FLORETS computes an exact alignment of client updates via distributed QR decomposition of local LoRA adapters thereby eliminating KnOTS inefficiency of needing the dense concatenation of full local model updates. This reduces server-side costs by over three orders of magnitude compared to KnOTS in each round. We share that the computational cost in FLORETS server is 14.25 GFLOP vs ~71 TFLOP in KnOTS for Llama-3.2-3B, K=10K=10, rk=16r_k=16.

  2. FLORETS introduces a data-free consensus merge rule. In contrast, KnOTS merges with TIES whose trimming hyperparameters are tuned per task on validation data, whereas a federated server holds none. Furthermore, the rank of the merged model update in KnoTS grows with the number of tasks, as a result of which it cannot serve as a compact broadcast in FL during every round. To mitigate these issues, FLORETS is designed to not rely on validation data and produces memory-efficient broadcast adapters by using spectral rank selection (ScreeNOT) followed by norm matching. Table 3 reports that removing rank selection and norm matching degrades the FLORETS performance by 2.71 points and 14.91 points, respectively. As an additional experiment, we find that KnOTS + TIES merging degrades the fine-tuning accuracy of Llama-3.2-1B on Dolly (non-iid, α=0.1\alpha=0.1) from 34.36% (FLORETS) to 25.05% (KnOTS + TIES) - justifying the novelty of FLORETS for FL setups.

We hope this addresses the concerns raised.

AuthorsOpenReview ↗

Response to W1 and W4: Where the effectiveness of FLORETS comes from, and why consensus merge helps

W1: Steps 1–3 vs. consensus merge, and FFA-LoRA + consensus merge

The gains come from Steps 1–3 and Step 4 together, and neither is effective on its own. Without alignment, a coordinate-wise sign vote is not meaningful: when both LoRA factors are trained, each client's factors can be rotated or rescaled without changing its update, so the sign of any entry is arbitrary. Without the consensus merge, alignment cannot change the aggregate, because a weighted sum is invariant to the choice of basis (Eq. 4).

(Additional Runs) The suggested FFA-LoRA + consensus merge experiment confirms this and further demonstrates the effectiveness of FLORETS. FFA-LoRA freezes a random A shared by all clients, so the vote is well-defined, but its basis is fixed in advance rather than built from the client updates.

MethodDolly (MMLU)GSM8K
FFA-LoRA30.6721.08
FFA-LoRA + consensus merge30.3721.68
FLORETS34.3624.03

The consensus merge barely changes FFA-LoRA (−0.30 and +0.60), whereas in the FLORETS basis it adds 4.57 points (Table 3). Our ablations in the response to Reviewer Rbit (W3) show the converse: with the merge held fixed, replacing the FLORETS basis with no basis, the vertical-stack basis or a single client's basis lowers accuracy to 26.67, 21.02 and 17.06, respectively.

W4: Further justification for the consensus merge

Under weighted averaging, opposing contributions cancel and shrink the direction that most clients agree on, so the minority signal is not preserved either. The consensus merge instead keeps the majority direction intact, returns exactly the weighted sum wherever clients agree (Remark 1), and outvotes a client only when the opposing side carries more weight, not by head count. Norm matching then restores the step size of the exact aggregate, and removing it lowers accuracy by 14.91 points (Table 3).

(Additional Analysis) The election discards little of what each client contributes: by the final round, the broadcast adapter retains 98–99% of each client's update energy on average, and at least 97% for every client. At every round, this retention is higher than when the summed aggregate is truncated to the same rank, as in FLoRIST and SpecTraL.

We hope this addresses the concerns raised. We will add the FFA-LoRA + consensus merge results to the ablation study in Sec. 4 (Table 3), and the per-client energy retention analysis to the appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for raising score to (3,3,3) + Additional discussion on Theory for final version

We sincerely thank all three reviewers for raising their scores: Reviewer Ti59 from 2 to 3, Reviewer WBEv from 1 to 3 together with the soundness and informativeness ratings from 1 to 2 each, and Reviewer Rbit from 2 to 3.

To further strengthen the manuscript for the final version, as suggested by Reviewer Ti59, we have prepared a theoretical analysis of the shared basis and the consensus merge of FLORETS. We would be grateful for your feedback on whether these additions would strengthen the work for the community and should be included in the final version. All measurements come from runs already reported in the paper. Throughout, τk\tau_k denotes the coordinates of client kk in the shared basis U^\widehat{U}, so that pkΔWk=U^τkp_k \Delta W_k = \widehat{U} \tau_k.

(Proposed for Main Paper, Sec. 3.1, after the current Observation, renamed Observation 1) Observation 2. Once the client updates are written in a common basis, their coordinates can be compared entry by entry. We call a coordinate contested when the clients do not share the same sign. Under severe heterogeneity, about 91% of coordinates are contested, both in our concatenation basis and in the vertical-stack basis, although they carry less than 30% of the total mass. The two bases differ in how decisively the conflict can be resolved. In the concatenation basis, the minority holds 35–37% of the contested mass, so the vote has a clear winner; in the vertical-stack basis, it holds 48–49%, close to a coin flip. This motivates both the consensus merge and the basis in which it operates.

(Proposed for Main Paper, Sec. 3.2, end of Step 3) Proposition 2. The shared basis is the principal basis of the client updates: (i) its columns are the eigenvectors of ∑kpk2ΔWkΔWk⊤\sum_k p_k^2 \Delta W_k \Delta W_k^{\top}; (ii) its leading jj columns capture the maximum total client energy over all rank-jj projections; and (iii) ∑kτkτk⊤=ΣC2\sum_k \tau_k \tau_k^{\top} = \Sigma_C^2 is diagonal, and for distinct singular values U^\widehat{U} is the only orthonormal basis of its span with this property, up to the signs and order of its columns. Part (iii) separates our basis from the vertical-stack basis, which spans the same subspace.

(Proposed for Main Paper, Sec. 3.2, replacing the current motivation of Step 4) Motivation. In the shared basis, coordinate (i,j)(i,j) denotes the same direction for every client, so client contributions can be compared entry by entry. Observation 2 shows that they frequently disagree in sign, and under a weighted sum each dissenting minority cancels part of the direction that most of the federation supports. We therefore resolve conflicts coordinate by coordinate, keeping the direction carried by the larger mass and removing the contributions that oppose it. A weighted sum cannot do this: it is invariant to the choice of basis, so it returns the same aggregate whether or not the updates are aligned.

(Proposed for Main Paper, Sec. 3.2, end of Step 4, replacing Remark 1) Proposition 3. Let s=∑kτks = \sum_k \tau_k be the coordinates of the exact aggregate, cc those of the consensus before norm matching, DijD_{ij} the mass discarded at coordinate (i,j)(i,j), and ρ=∥s∥F/∥c∥F\rho = \lVert s \rVert_F / \lVert c \rVert_F. Then (i) cij=sij+sign(sij)Dijc_{ij} = s_{ij} + \mathrm{sign}(s_{ij}) D_{ij}, so the consensus keeps every elected sign and returns the federated average exactly wherever the clients agree; (ii) cos⁡(c,s)≥ρ\cos(c,s) \ge \rho; and (iii) the norm-matched update deviates from the federated average by at most 2(1−ρ)∥ΔW∗∥F22(1-\rho)\lVert \Delta W^{\ast} \rVert_F^2.

(Proposed for Appendix C, new subsection "Theoretical Analysis") Supporting analysis and proofs. This subsection would contain:

  • a table of the sign-election statistics behind Observation 2;
  • an amplification analysis: at a coordinate with minority share mm, the consensus scales the aggregate by (1−m)/(1−2m)(1-m)/(1-2m), about 2.2–2.4 in our basis and 14–26 in the vertical-stack basis, consistent with the basis ablation of Table 3;
  • Corollary 1: ∥c∥F≥∥ΔW∗∥F\lVert c \rVert_F \ge \lVert \Delta W^{\ast} \rVert_F, so norm matching always scales the consensus down, by the factor ρ≤1\rho \le 1, and restores the step size of the federated average;
  • the measured ρ\rho: in the first round on Dolly / Llama-3.2-1B (α=0.1\alpha = 0.1), ρ\rho is at least 0.81 on every adapted matrix (mean 0.89), so the consensus direction stays within about 36° of the federated average;
  • the proofs of Propositions 2 and 3 and Corollary 1.

(Proposed for Appendix C, end of the Norm Matching subsection) A one-sentence pointer to Corollary 1.

We would be grateful for your suggestions if our above due diligence helps further improve the manuscript for the community. Accordingly, we will include it in the final version. Thank you again for your constructive reviews.

The exact proposed text of all additions, including the full statements and proofs, is available at: https://lavender-deloris-95.tiiny.site

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.