acceptodds
Withdrawn from ICLR 2027

FedRRA: Residual Rank Adaptation for Heterogeneous Federated Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Combining Large Language Models (LLMs) with Low-Rank Adaptation (LoRA) and Federated Learning (FL) has emerged as a promising paradigm to enhance Parameter-Efficient Fine-Tuning (PEFT) while preserving data privacy. However, in practical deployments, where clients have heterogeneous resources and thus require different LoRA ranks for fine-tuning, the conventional paradigm of aggregating uniform LoRA adapters is no longer applicable, introducing significant challenges. While padding-based methods, such as zero-padding, have been proposed to address this issue, we find that they still suffer from two key challenges: limited adaptability to heterogeneous resource settings and aggregation noise. To address these challenges, we propose Federated Residual Rank Adaptation (FedRRA), a novel heterogeneous Federated Fine-Tuning (FFT) framework. Specifically, we design a new padding strategy, termed Consensus-Aware Padding (CAP), which performs weighted padding by combining rank-one signal strength with direction-aware soft gating, enabling improved adaptation to diverse resource-heterogeneous scenarios. In addition, we propose a residual correction method tailored for heterogeneous resource settings, which mitigates aggregation noise by adding a residual term to the frozen pretrained weights. Extensive experiments demonstrate that FedRRA consistently outperforms existing methods across different resource-heterogeneous settings.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, Large Language Models, LoRA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5PJXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses federated LoRA fine-tuning when clients use different ranks (4 to 64). Building on the pad-to-maximum-rank, average and truncate framework, it proposes FedRRA with two parts:

CAP (Consensus-Aware Padding) fills a low-rank client's missing k-th dimension with a weighted average of the k-th components of clients that have it. A client's weight is the magnitude of its component times how much its direction agrees with the other clients, and only clients with above-average agreement contribute. ARC (Aggregation Residual Correction) has the server compute the difference between the average of the clients' BA products (the ideal update) and each client's truncated LoRA product, compress it with a rank-16 SVD, and send it to the client to add to its frozen weights.

On TinyLlama, Qwen2.5-3B and Llama2-7B fine-tuned on Dolly, Alpaca and Wizard, FedRRA scores higher than the baselines on MMLU and MT-Bench in every setting.

Critical strengths
  1. Heterogeneous client ranks are a practical problem in federated fine-tuning, and padding-based methods are simple and cheap.
Critical weaknesses
  1. The improvements are small and the evaluation discriminates poorly. Only three seeds are used and no significance tests are reported. Base-model scores before fine-tuning are not reported, so it is impossible to tell how much fine-tuning changes the metrics.
  2. The server cost is high. The introduction criticizes FlexLoRA for large-scale SVD with extra computation and decomposition error, yet FedRRA performs an SVD with truncation for every client's matrices in every round, which also incurs truncation error. In Table 16, FedRRA's server time per round on TinyLlama is 27.78 s against 0.51 s for HETLoRA and 17.89 s for FlexLoRA, and its communication (92.36M parameters) exceeds that of HETLoRA and FlexLoRA (69.84M). So the motivation conflicts with the method.
  3. The models used in the experiments are old. The setting is narrow. Training runs for only 4–5 rounds with 10 clients, and LoRA is applied only to q and v.
Questions for authors
  1. Can you use more advanced models?
  2. Can you add homogeneous baselines in which all clients use rank, and exact aggregation combined with FLoRA or FlexLoRA?
  3. Can you add seeds and run significance tests?
  4. The server performs SVDs for every client's matrices each round. How does server time grow for models beyond 7B and for hundreds of clients?
Reviewer_RGi4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedRRA, a framework for federated fine-tuning of large language models with heterogeneous LoRA ranks, which addresses ineffective padding strategies under heterogeneous client resources and aggregation errors caused by independently averaging LoRA factors. FedRRA introduces Consensus-Aware Padding (CAP) and Aggregation Residual Correction (ARC). The experiments demonstrate consistent improvements over several federated LoRA baselines.

Critical strengths
  1. Comprehensive experimental evaluation across heterogeneous settings.

  2. The paper identifies two challenges in heterogeneous federated LoRA fine-tuning and addresses them within a unified framework.

Critical weaknesses
  1. CAP computes directional consensus by comparing corresponding columns of the LoRA B matrices across clients (Equations 6–8). However, different pairs of A and B matrices can represent exactly the same weight update, which means LoRA factorization is not unique. Consequently, corresponding rank-one components across independently trained clients are not necessarily aligned, and cosine similarity between B columns does not necessarily reflect agreement between their underlying weight updates. It's not sufficiently explain how cross-client component correspondence is maintained.

  2. Although the paper introduces residual correction to mitigate aggregation noise, it does not clearly formalize the remaining aggregation error after applying the proposed method with SVD compression. Equations 13–14 suggest that residual correction can exactly eliminate aggregation errors, but the actual implementation uses rank-16 truncated SVD, which inevitably introduces approximation errors unless the residual is sufficiently low-rank.

  3. Figures 9 and 10 evaluate aggregation errors primarily using the Frobenius norm of the difference between weight updates. Although this metric measures parameter-level discrepancies, it does not necessarily reflect their impact on model outputs or downstream performance.

Questions for authors

See weaknesses.

Reviewer_Yip7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FedRRA addresses federated LoRA fine-tuning when clients train adapters of different ranks. It builds on padding-based aggregation and has two parts.

  • Consensus-Aware Padding (CAP) fills the missing rank dimensions of low-rank clients with a weighted combination of the clients that trained each dimension. The weight multiplies the rank-one signal strength ∥bk∥2∥ak∥2\|b_k\|_2\|a_k\|_2 by a mean-referenced soft gate on the cosine similarity between a client's BB column and the consensus of the other clients.
  • Aggregation Residual Correction (ARC) extends the residual idea of FedEx-LoRA. The server computes the native correction target ΔWnat=∑ipiBiAi\Delta W_{nat} = \sum_i p_i B_i A_i and truncates the averaged padded adapters to each client's rank. It then sends each client a residual ΔWresi\Delta W_{res_i}, which is added to the client's frozen W0W_0. Residuals are compressed by truncated SVD with rank 16.

Experiments use TinyLlama, Qwen2.5-3B and Llama2-7B with ten clients, train on Dolly, Wizard and Alpaca, and evaluate on MMLU and MT-Bench under the Triangle, Diamond and Head rank allocations. FedRRA reports the highest federated mean in every setting. The appendix adds 50 clients, Dirichlet α=0.1\alpha = 0.1, component ablations and server cost measurements.

Critical strengths
  1. FedRRA ranks first across many federated settings. Table 2 and Tables 8 to 10 cover three backbones, three rank allocations, three training datasets and MT-Bench, and Tables 11 and 12 add 50 clients and Dirichlet α=0.1\alpha = 0.1. FedRRA has the highest federated mean in all 48 entries, while the strongest competing method changes from setting to setting (FedPLoRA in 29 entries, FlexLoRA in 9, FLoRA in 7, FedRep in 3). A uniform ranking over this many entries is harder to explain by seed noise than any single margin. This is the main reason I take the empirical result seriously, although each margin is small (Weakness 2).

  2. The ablations test the actual design choices, and the cost of the method is reported openly.

    • Table 3 separates CAP from ARC.
    • Table 4 compares the native-rank target and client-specific residuals with ARC-PT and ARC-SR.
    • Table 5 and Figure 6 compare the mean-referenced soft gate with "w/o Gate", "Zero-Ref" and fixed thresholds.
    • Table 13 compares BB, AA and BABA as the consensus direction, and Table 25 compares the Frobenius score with gradient, Taylor and Fisher scores.
    • Figures 20 and 21 sweep the truncation rank rsvdr_{svd}.
    • Most of these are repeated on Llama2-7B (Tables 20 to 22).

    Tables 16 and 17 show that FedRRA has the highest server time and GFLOPs of all methods (27.78 s per round on TinyLlama, against 0.51 s for HETLoRA and 17.89 s for FlexLoRA). A reader can therefore see the trade-off between communication and server computation.

Critical weaknesses
  1. ARC's exact aggregation is shown for one round, and the multi-round procedure in Algorithm 1 does not preserve it when ranks differ.

    • The extra term. Eq. 14 shows that after one correction, client ii's effective weight is W0j+ΔWnatW_0^j + \Delta W_{nat}. In Algorithm 1, each client adds ΔWresi\Delta W_{res_i} to its own frozen W0W_0 (line 5) and starts the next round from its truncated adapters BiAi\mathbf{B}_i\mathbf{A}_i (line 6). The next native target ∑kpkBkAk\sum_k p_k B_k A_k therefore contains ∑kpkBkAk\sum_k p_k \mathbf{B}_k\mathbf{A}_k, while client ii's frozen weight removes only its own BiAi\mathbf{B}_i\mathbf{A}_i. Writing out two rounds, client ii's new effective weight equals its previous one, plus the weighted average of the local changes, plus ∑kpkBkAk−BiAi\sum_k p_k\mathbf{B}_k\mathbf{A}_k - \mathbf{B}_i\mathbf{A}_i.
    • What it does. The last term is zero when all ranks are equal, which is the FedEx-LoRA case. Under heterogeneous ranks it is nonzero and differs across clients. Its pp-weighted sum is zero, so the weighted mean of the client models follows the ideal update, but the individual client models drift apart and train from different starting points. A simple check: if no client changes its adapters in a round, every client model should stay the same, but under Algorithm 1 each one moves by this term.
    • The compression is unclear. Eq. 15 applies rank-16 SVD to the server residual ΔWres\Delta W_{res}, but clients receive ΔWresi\Delta W_{res_i}. Neither algorithm contains a compression step, and Table 15 counts one SVD per layer rather than one per client residual.

    Exact aggregation is the paper's explanation for ARC, so this gap affects the main contribution.

  2. The margins are small, several numbers disagree with each other, and the evidence does not tie the gains to exact aggregation.

    • The margins are within noise. In Table 2, FedRRA leads the second-best method by 0.29 to 0.43 MMLU points and by 0.10 to 0.11 MT-Bench points, while the reported ± values reach 0.35. TinyLlama's MMLU scores in Table 2 and Tables 9 to 12 lie between 24.45 and 26.95, at the 25% chance level of four-option MMLU, and the main-text ablations (Tables 3 to 5) use TinyLlama.
    • Appendix D.6 disagrees with the main tables. In its text, FedRRA on TinyLlama peaks at 25.34 on Dolly and 25.69 on Wizard, against 26.37 and 25.96 in Table 2. FlexLoRA plateaus at 25.62 on Wizard, only 0.07 below FedRRA's peak, against a 0.87-point gap in Table 2. On Qwen2.5-3B, FedRRA peaks at 65.98 on Dolly and 65.86 on Wizard, against 66.89 and 66.49 in Table 8, and FlexLoRA peaks at 65.88 on Wizard, above FedRRA. These differences exceed the margins over the baselines.
    • Figure 9b does not test the mechanism. The near-zero error after correction follows from the definition of ΔWresi\Delta W_{res_i} when the residual is not truncated, and Appendix F.5 reports that truncated residuals often match or beat the full residual.
    • The padding gap is not explained. Within one uncompressed round, exact ARC makes the corrected model independent of the padding. Yet the gap between CAP and Zero-Padding becomes larger once ARC is added, in every column of Tables 3, 18, 19 and 20 (for example from 0.31 to 0.80 MMLU points on Wizard in Table 3). Whatever produces this gap must act through the next round's starting adapters, the drift in Weakness 1 or the rank-16 truncation, and the paper analyzes none of these.

    The source of the gains is therefore not established.

Questions for authors
  1. What multi-round recurrence does the code implement?

    • Is each client's frozen W0W_0 updated cumulatively as in Algorithm 1, line 5, and does each client start the round from its truncated adapters?
    • With uncompressed residuals, do all client models stay identical and unchanged over two rounds in which no client changes its adapters?
    • If the implementation differs from Algorithm 1, which results use it?
    • What are the "global model" and the per-rank "local models" in Figure 4, and which model is evaluated in Table 2, given that each client holds its own W0W_0?

    Using the target ∑kpkδk+BiAi\sum_k p_k \delta_k + \mathbf{B}_i\mathbf{A}_i, where δk\delta_k is client kk's local change in the round, would keep every client at the same exact model, and the server already holds these quantities. An implementation with this property would resolve Weakness 1 and raise my soundness score.

  2. Which matrix does the rank-16 SVD compress in the main experiments: ΔWres\Delta W_{res}, each ΔWresi\Delta W_{res_i}, or another construction?

    • How many SVDs does the server compute per round, and how does this match Table 15? The server time grows only from 27.78 s to 43.11 s between 10 and 50 clients (Tables 16 and 17).
    • Was Figure 9b computed with the compressed residual?
    • Each client also keeps and updates a private dense copy of W0W_0. What is its memory and storage cost on the client side?
  3. Which MMLU protocol do the main experiments use: the full test set or the 1,444-question subset of Appendix C.2, and with how many shots?

    • Which model judges MT-Bench, and what does ± denote?
    • What does each backbone score before fine-tuning?
    • Why do the per-round values in Appendix D.6 stay below the corresponding entries in Tables 2 and 8, and why does the ranking against FlexLoRA differ? Were the reported rounds selected on the evaluation data?

    If the tables and Appendix D.6 come from different protocols, the margins in Table 2 should be read against that difference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.