acceptodds
Under review as a conference paper at ICLR 2027

Preventing Rank Collapse in Federated Low-Rank Adaptation with Client Heterogeneity

Abstract

Federated low-rank adaptation (FedLoRA) has facilitated communication-efficient and privacy-preserving fine-tuning of foundation models for downstream tasks. In practical federated learning scenarios, client heterogeneity in system resources and data distributions motivates the use of heterogeneous LoRA ranks across clients. However, we identify a previously overlooked phenomenon in heterogeneous FedLoRA with SVD-based allocation, termed rank collapse, where the energy of the global update becomes concentrated in the minimum shared rank, resulting in suboptimal performance and high sensitivity to rank configurations. Through theoretical analysis, we reveal the root cause of rank collapse: a mismatch between rank-agnostic aggregation weights and rank-dependent client contributions, which systematically suppresses higher-rank updates at a geometric rate over rounds. Motivated by this insight, we propose raFLoRA, a rank-partitioned aggregation method that decomposes local updates into rank partitions and then aggregates each partition weighted by its effective client contributions. Extensive experiments across vision, language, and reasoning tasks show that raFLoRA prevents rank collapse, improves model performance, and enhances robustness across diverse heterogeneous configurations compared with strong FedLoRA baselines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning, LoRA, Heterogeneous Rank

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_twrbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the rank collapse phenomenon in heterogeneous rank FedLoRA, pointing out that uniform aggregation weights weaken high-rank components that are only supported by a subset of clients. The authors analyze the cumulative process of this attenuation under simplified assumptions and propose raFLoRA, which employs normalization based on the clients effectively contributing to the components. Experiments across vision, language, and reasoning tasks validate the method's effectiveness.

Critical strengths
  1. The problem analysis is valuable. The paper links high-rank energy decay with client coverage, providing a more explanatory analysis than simply comparing accuracy.

  2. The method is simple and is well supported by experiments. The results demonstrate the practical application of high-rank components without increasing client communication volume.

Critical weaknesses
  1. The conclusions regarding geometric decay rely on strong assumptions, such as fixed singular directions and the absence of singular value crossings. The appendix discusses basis changes, but it has not yet proven that rank collapse still occurs under perturbations, nor has it quantified the degree of deviation of actual training from these assumptions. Therefore, the theory's explanation of real training remains limited.

  2. The gains on some tasks are modest. For example, the average accuracy of commonsense reasoning of the 8B model is only 0.27 percentage points higher than that of FlexLoRA.

Questions for authors
  1. Could you provide data on the evolution of singular subspaces or instances of singular value crossing during actual training to illustrate the applicability of the stable spectral routing hypothesis?

  2. For inference tasks where performance gains are modest, are the overall inference benefits stable?

4 comments

We thank the reviewer for recognizing the value of our analysis and the effectiveness of raFLoRA. To address the reviewer's remaining questions and concerns, we provide the following clarifications.

W1 & Q1: The gap between theory and actual training

Theoretical scope. Appendix C relaxes the stable spectral routing assumption and derives a one-round perturbation bound. Extending this analysis across rounds is challenging. Specifically, singular-value crossings can move directions between partitions with different contributor coverage, while basis drift and mixing can redistribute energy across partitions. These effects couple the partition energies and make their evolution depend on the training trajectory, complicating the derivation of a closed recursion. A multi-round guarantee would require controlling the cumulative perturbations tightly enough for the retention gap to continue driving contraction. Establishing when rank collapse persists under such perturbations is a nontrivial extension that we leave to future work.

The one-round contributor dilution in Eq. (3) follows directly from the aggregation rule and holds independently of stable spectral routing. Because higher-rank partitions have fewer contributors, their aggregated updates may be more sensitive to client participation and alignment. Under non-IID training, weaker alignment may limit the accumulation of higher-rank contributions across rounds, compounding the dilution caused by uniform averaging. In Fig. 5(b), FlexLoRA exhibits the spectral signature of rank collapse under non-IID training. Its higher-rank energy fraction decreases further as data heterogeneity increases.

Empirical assessment. To assess how closely stable spectral routing approximates actual training, we measure singular-subspace overlap between consecutive rounds and compare one-step estimates of the higher-rank energy fraction with observed values. The shared top-8 subspace has a mean overlap of 98.07% under IID training and 97.36% under non-IID training. Across the tracked layers, the one-step model reduces squared error relative to a baseline that uses each layer’s mean observed fraction by an average of 86.27% under IID training and 84.57% under non-IID training. These results support stable spectral routing as a local approximation for the shared low-rank subspace. The lower overlap for larger subspaces suggests that extending the theory to higher ranks may require accounting for subspace drift.

We evaluate FlexLoRA on CIFAR-100 with ViT-Base, using 100 clients and sampling 10 per round. Client ranks are uniformly distributed over {8,16,32,48,64}\lbrace 8,16,32,48,64\rbrace. We run 100 rounds under IID and Dirichlet non-IID (α=0.1\alpha=0.1) partitions with matched seeds and track the query projection updates in layers 0, 6, and 11.

1) Subspace evolution. Let the aggregated update at layer ℓ\ell and round tt be ΔWℓ(t)=Uℓ(t)Σℓ(t)(Vℓ(t))⊤\Delta W_\ell^{(t)}=U_\ell^{(t)}\Sigma_\ell^{(t)}(V_\ell^{(t)})^\top. We measure the normalized overlap between consecutive cumulative top-rr subspaces as Oℓ,r(t)=12r(∥(Uℓ,r(t−1))⊤Uℓ,r(t)∥F2+∥(Vℓ,r(t−1))⊤Vℓ,r(t)∥F2)O_{\ell,r}^{(t)}=\frac{1}{2r}\left(\lVert(U_{\ell,r}^{(t-1)})^\top U_{\ell,r}^{(t)}\rVert_F^2+\lVert(V_{\ell,r}^{(t-1)})^\top V_{\ell,r}^{(t)}\rVert_F^2\right). This projector-based metric is invariant to sign changes and rotations within the subspace, and O=1O=1 indicates identical consecutive subspaces. The table reports the mean overlap across the three tracked layers and all 99 adjacent-round transitions.

Top-rr subspaceIIDnon-IID
898.07%97.36%
1690.24%88.37%
3279.29%78.71%
4877.16%76.73%
6476.65%76.57%

The shared top-8 subspace has a mean adjacent-round overlap of 98.07% under IID data and 97.36% under non-IID data. Overlap decreases as rr increases, and the two settings differ by at most 1.87 percentage points. These results suggest that stable spectral routing is an accurate local approximation for the shared low-rank core, but a weaker approximation for higher-rank directions.

2) One-step fit of the higher-rank energy fraction. We define the observed direction energy as ei,ℓ(t)=(σi,ℓ(t))2e_{i,\ell}^{(t)}=(\sigma_{i,\ell}^{(t)})^2 and the higher-rank energy fraction as yℓ(t)=1−ρ8,ℓ(t)=∑i=964ei,ℓ(t)∑i=164ei,ℓ(t)y_\ell^{(t)}=1-\rho_{8,\ell}^{(t)}=\frac{\sum_{i=9}^{64}e_{i,\ell}^{(t)}}{\sum_{i=1}^{64}e_{i,\ell}^{(t)}}. For a client supporting direction ii, we measure βk,i,ℓ(t)=(ui,ℓ(t))⊤ΔWk,ℓ(t)vi,ℓ(t)/σi,ℓ(t)\beta_{k,i,\ell}^{(t)}=(u_{i,\ell}^{(t)})^\top\Delta W_{k,\ell}^{(t)}v_{i,\ell}^{(t)}/\sigma_{i,\ell}^{(t)}. Let μi,ℓ(t)\mu_{i,\ell}^{(t)} and νi,ℓ(t)\nu_{i,\ell}^{(t)} be its empirical first and second moments, and let pi=Pr⁡(rk≥i)p_i=\Pr(r_k\geq i). Under the local fixed-subspace approximation, the expected one-step contraction factor is qi,ℓ(t)=(1−τ)pi2(μi,ℓ(t))2+τpiνi,ℓ(t)q_{i,\ell}^{(t)}=(1-\tau)p_i^2(\mu_{i,\ell}^{(t)})^2+\tau p_i\nu_{i,\ell}^{(t)}, where τ=(K−M)/(M(K−1))\tau=(K-M)/(M(K-1)). This gives e^i,ℓ(t+1)=qi,ℓ(t)ei,ℓ(t)\widehat e_{i,\ell}^{(t+1)}=q_{i,\ell}^{(t)}e_{i,\ell}^{(t)} and y^ℓ(t+1)=∑i=964e^i,ℓ(t+1)∑i=164e^i,ℓ(t+1)\widehat y_\ell^{(t+1)}=\frac{\sum_{i=9}^{64}\widehat e_{i,\ell}^{(t+1)}}{\sum_{i=1}^{64}\widehat e_{i,\ell}^{(t+1)}}.

Over the valid adjacent-round transitions, we report MAE=1n∑∣y−y^∣MAE=\frac{1}{n}\sum|y-\widehat y|, RMSE=1n∑(y−y^)2RMSE=\sqrt{\frac{1}{n}\sum(y-\widehat y)^2}, and R2=1−∑(y−y^)2∑(y−yˉ)2R^2=1-\frac{\sum(y-\widehat y)^2}{\sum(y-\bar y)^2}, where yˉ\bar y is the mean of the observed fractions over the valid transitions. Lower MAEMAE and RMSERMSE indicate smaller fitting errors, while a higher R2R^2 indicates a better fit relative to the mean baseline.

PartitionLayerMAERMSER2R^2
IID00.01960.04110.8312
IID60.01840.04020.8318
IID110.00610.01740.9252
IIDLayer Average0.01470.03290.8627
non-IID00.01660.03760.8101
non-IID60.01270.03100.8363
non-IID110.00580.01470.8908
non-IIDLayer Average0.01170.02780.8457

The consistently high one-step R2R^2 values show that the approximation captures local energy evolution under both IID and strongly non-IID data. In particular, non-IID partitioning only slightly reduces the averaged R2R^2 from 0.8627 to 0.8457. The deepest Layer 11 yields the best fit in both settings, with R2R^2 values of 0.9252 and 0.8908, respectively. This stronger fit may reflect lower subspace drift and greater concentration of update energy in the dominant singular directions, which could reduce the cross-direction mixing omitted by the stable spectral routing assumption.

W2 & Q2: Magnitude and stability of performance gains

In the main experiments, we report accuracy as mean ± standard deviation across three random seeds. With LLaMA-3.1-8B, raFLoRA shows lower run-to-run variability than FlexLoRA. On GSM8K, the reported variability is ±0.27 for raFLoRA and ±0.74 for FlexLoRA. For the average across eight commonsense reasoning tasks, the corresponding values are ±0.08 for raFLoRA and ±0.67 for FlexLoRA.

Table 11 in Appendix H reports results for each of the eight commonsense reasoning subtasks. raFLoRA outperforms FlexLoRA on five subtasks, with the largest accuracy gains on WG (+1.18), ARC-c (+1.17), and SIQA (+0.95). The results in Sections 6.5 and 6.6, together with the additional experiments in Appendix E, further show that raFLoRA improves performance under stronger data heterogeneity and across a range of client rank configurations and distributions.

We have added the experimental results described above to the revised manuscript, with the new text shown in dark purple.

In closing, we would like to once again thank the reviewer for the insightful comments. We hope that our responses have satisfactorily addressed the remaining concerns.

Thank you for the clarification. My concerns have been addressed. I will maintain the original rating.

We sincerely thank the reviewer for maintaining the positive evaluation. We are pleased that our responses have addressed your concerns.

Reviewer_r1FZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper identifies an issue in heterogeneous-rank federated low-rank adaptation where most of the singular-value energy becomes concentrated in the minimum shared rank and thus limits the benefits of higher-rank adaptation. The authors propose raLoRA which performs rank-dependent aggregation. Empirical studies show raLoRA prevents rank collapse and improves model performance across vision, language, and reasoning tasks.

Critical strengths
  1. The rank collapse phenomenon under FedAvg mechanism is grounded by theoretical studies and the proposed aggregation method is simple and effective in directly addressing the dilution and has an intuitive interpretation.

  2. The authors do empirical studies on multiple model architectures and downstream tasks, and the reported improvements are generally consistent across the experiment setups.

Critical weaknesses
  1. raLoRA's core idea of correcting the aggregation denominator according to effective contributors has precedents. HeteroFL[1] proposes to aggregates heterogeneous model components only across clients containing the corresponding parameters. HetLoRA[2] also considers rank heterogeneity and sparsity-aware aggregation. To be fair, raLoRA brings the idea to SVD-based LoRA allocation, which is unique as far as I know, but the principal algorithmic contribution appears incremental.

[1] Diao, E., Ding, J. and Tarokh, V., 2020. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264.

[2] Cho, Y.J., Liu, L., Xu, Z., Fahrezi, A. and Joshi, G., 2024, November. Heterogeneous lora for federated fine-tuning of on-device foundation models. In Proceedings of the 2024 conference on empirical methods in natural language processing (pp. 12903-12913).

Questions for authors
  1. The experimental setup appears to use a fixed initial learning rate with linear decay across all methods and tasks, as specified in Section 6.1, which leaves the learning rate sensitivity not analyzed. This is especially concerning as raFLoRA changes the magnitude of global updates. A learning rate sensitivity analysis will make the reported empirical findings more convincing.

  2. raFLoRA gives higher-rank clients exclusive influence over additional rank partitions. This can be interpreted as correcting contributor dilution, but it also creates a potential source of bias when client rank correlates with local data distribution or quality. Can the authors evaluate how much empirical gains come from such bias?

  3. Figure 5(b) shows that raFLoRA maintains relatively similar higher-rank energy ratios across different levels of data heterogeneity, whereas Figures 2(c) and 6(a) indicate that downstream accuracy remains substantially sensitive to the degree of heterogeneity. This discrepancy suggests that preserving higher-rank spectral energy may not be sufficient to explain the observed performance improvements.

3 comments

We thank the reviewer for recognizing our theoretical analysis, the effectiveness of our method, and the consistent empirical improvements across model architectures and downstream tasks. We address the reviewer's concerns and questions through the following clarifications.

W1: Relation to HeteroFL and HetLoRA

We thank the reviewer for highlighting HeteroFL and HetLoRA as relevant prior work. We have discussed HetLoRA in the Related Work section and have included it as a baseline in our main experiments. HeteroFL addresses heterogeneity in model width and thus falls outside the scope of our experimental comparison in heterogeneous-rank FedLoRA. We compare the aggregation mechanisms of HeteroFL, HetLoRA, and raFLoRA below to clarify our contributions.

In HeteroFL, clients train nested subnetworks of different widths, and the server aggregates each weight coordinate only over clients whose subnetworks contain it. In heterogeneous-rank FedLoRA, the client’s LoRA rank determines the inner dimension of BkAkB_kA_k, but every reconstructed update ΔWk=BkAk\Delta W_k = B_kA_k has the same shape as the full weight matrix. Applying HeteroFL’s coordinate-wise rule directly to these updates therefore uses the same set of participating clients at every coordinate. The result is uniform averaging of the reconstructed updates, which cannot distinguish which clients contribute to each rank partition.

HetLoRA assigns each client a single weight based on the Frobenius norm of its reconstructed update and uses that weight to aggregate the zero-padded AA and BB factors separately. The weight reflects the overall magnitude of the client’s update and is shared across rank partitions, leaving differences in contributor coverage uncorrected. Meanwhile, multiplying the separately aggregated LoRA factors introduces cross-client terms.

In contrast, raFLoRA aggregates within-client LoRA products by rank partition and normalizes over each partition’s contributors, correcting dilution from unequal coverage and avoiding cross-client terms. Beyond the aggregation rule itself, we identify a previously overlooked rank collapse phenomenon in SVD-based heterogeneous-rank FedLoRA, derive the one-round dilution effect, and show that under stable spectral routing, the higher-rank energy fraction has a geometrically decaying upper bound. Experiments with multiple foundation models on diverse downstream tasks show that raFLoRA mitigates spectral energy concentration and yields more consistent downstream performance gains across data heterogeneity settings and client rank configurations.

Q1: Learning rate sensitivity analysis

Following the reviewer's suggestion, we evaluate learning rate sensitivity using Llama-3.2-3B on GSM8K. Given our computational budget, we limit the sweep to learning rates from 1e-4 to 1e-3 in increments of 1e-4, keep all other settings identical to those in the corresponding main experiment. All results are test accuracies (%).

Learning RateFlexLoRAraFLoRA
1e-432.3734.04
2e-436.4738.89
3e-437.9841.02
4e-439.5840.03
5e-439.0441.62
6e-439.4240.41
7e-439.1242.91
8e-440.4141.77
9e-440.0340.26
1e-339.5839.95

As shown in the table, accuracy varies with the initial learning rate for both methods, but raFLoRA outperforms FlexLoRA at all ten tested rates. The gains range from 0.23 to 3.79 percentage points and average 1.69 points. At their respective best tested rates, raFLoRA achieves 42.91% at 7e-4, while FlexLoRA achieves 40.41% at 8e-4, a gap of 2.50 percentage points. These results indicate that raFLoRA’s advantage over FlexLoRA on GSM8K is robust to the choice of initial learning rate within the tested range.

Q2: Potential bias from rank-data correlations

Following the reviewer's suggestion, we conduct two experiments with ViT on CIFAR-100 to assess whether raFLoRA’s gains over FlexLoRA depend on how client ranks are associated with local data distributions and quality. Both experiments use the same client data partitioning setup as the main experiment. All results are test accuracies (%).

1) Randomized rank assignment. We fix the data partition and the set of clients selected in each round, and vary only the assignment of ranks to clients across five seeds. This tests whether the gain depends on a particular pairing of ranks and local datasets.

MethodSeed 0Seed 1Seed 2Seed 3Seed 4Average
FlexLoRA83.4584.1283.6783.6283.6583.70
raFLoRA86.2085.3185.0685.0787.1585.76

As shown in the table, accuracy varies with how client ranks are paired with local datasets, but raFLoRA consistently outperforms FlexLoRA across all five assignments, with gains of 1.19-3.50 percentage points and an average gain of 2.06 points.

2) Opposite associations between client rank and data quality. We add Gaussian noise to the local data of either the lowest-rank or the highest-rank clients, with larger ν\nu indicating stronger noise. These two settings simulate positive and negative associations between client rank and data quality, respectively. Section 6.6 of the manuscript also describes the experiment with noise applied to the lowest-rank clients.

Clients receiving noiseMethodν=0.0\nu=0.0ν=0.1\nu=0.1ν=0.3\nu=0.3ν=0.5\nu=0.5
Lowest rankFlexLoRA83.9485.1582.6781.45
Lowest rankraFLoRA87.4387.4786.9186.61
Highest rankFlexLoRA83.9484.4082.8781.19
Highest rankraFLoRA87.4387.0586.6486.18

When Gaussian noise is applied to the highest-rank clients, raFLoRA still outperforms FlexLoRA by 2.65-4.99 percentage points across the nonzero noise levels. At each matched noise level, the gain over FlexLoRA differs by at most 0.47 percentage points between corrupting the lowest-rank and highest-rank clients. This suggests that raFLoRA’s gains do not depend on higher-rank clients having cleaner data.

Q3: Higher-rank energy preservation versus downstream performance

The similarity in Figure 5(b) concerns the fraction of total spectral energy retained in higher-rank components. Updates with similar energy ratios can still differ in magnitude, direction, and task relevance. They can therefore yield different downstream accuracies across heterogeneity settings, as observed in Figures 2(c) and 6(a).

One plausible explanation is that, under weaker data heterogeneity, clients’ local objectives tend to be more similar and their updates may be better aligned. As heterogeneity increases, updates may become more specific to local distributions, producing divergent or even conflicting directions. raFLoRA makes higher-rank capacity available for learning, but it does not eliminate other learning difficulties caused by data heterogeneity. Understanding how data heterogeneity continues to affect learning when higher-rank capacity is preserved remains an important question for future work. This weaker alignment may help explain why downstream accuracy declines as data heterogeneity increases, even when raFLoRA preserves higher-rank energy.

We have added the discussion and additional experimental results described above to the revised manuscript, with the new text shown in dark orange.

In closing, we sincerely thank the reviewer again for the constructive feedback. We hope that our clarifications and additional evidence have addressed the concerns raised and provided a stronger basis for assessing the contributions and effectiveness of our work.

Thank you again for your thoughtful review and constructive feedback. Given the limited time for discussion, we would appreciate hearing whether our response addresses your concerns. We would be happy to answer any remaining questions.

Reviewer_VwpDOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Empirically noted in previous literature on heterogenous FedLoRA is the phenomenon of rank collapse, where the updates of clients using lower LoRA update ranks dominate the dynamics of the aggregate fine-tuning process and diminish the contributions of clients with more capable hardware/systems who use higher, more accurate update ranks. The authors identify the theoretical cause of rank collapse and propose their own algorithm which solves the issue. Empirical verification shows that their algorithm remediates the effects of rank collapse.

Critical strengths
  1. The presentation of the paper is exceptionally clear. I am no expert in FedLoRA. However, I feel like I can follow the conceptual tasks of the paper almost completely, and even mostly follow the mathematical conventions, even upon a first reading.

  2. The authors present a very thorough analysis of the topic. The problematic phenomenon of rank collapse is described, and its empirical evidence noted in existing literature. Then, a theoretical analysis of its origin/cause is given. This alone would be a significant contribution to me. But they go a step further in designing an empirically verifying an algorithm which solves the problem.

  3. Ample empirical evidence backs up their claims as the reader is guided through the paper.

Critical weaknesses
  1. Some of the paper's sentences and terms are a little vague. For example, on p. 4 we read "Instead, the learning dynamics are governed by the shared rank r1, rendering higher-rank components progressively ineffective." It is unclear what exactly is meant by "learning dynamics" and "ineffective". To my understanding, the issue is that FedLoRA fine-tuning with heterogenous ranks across clients can be problematic when the top singular values of the lowest-rank clients dominate the top singular values of the final aggregate LoRA update across clients. Correct me if I'm wrong.
Questions for authors
  1. What is meant by "accuracy" near the bottom of p.1 of the paper? That is a vague term here. I also see a vague reference to accuracy on the top of p.7. And more references in some of the pages of the appendices.

  2. What distinguishes "algebraic rank" from "effective rank"? Why is it in issue to have high algebraic rank but low effective rank? Honestly just curious and trying to clarify what the paper is saying.

1 comment

We sincerely thank the reviewer for recognizing the clarity of our presentation, the depth of our analysis, and the strength of our empirical evidence. To address the reviewer's remaining questions, we would like to provide the following clarifications.

W1: Clarification of the rank-collapse mechanism

Yes, the reviewer's interpretation is correct. We thank the reviewer for pointing out the ambiguity in “learning dynamics” and “ineffective.” We have updated the sentence in blue on p.5 of the revised manuscript as follows:

“As FL rounds progress, the shared rank r1r_1 increasingly dominates the top singular values of the aggregated LoRA update, progressively diminishing the contribution of the additional components from higher-rank clients.”

This phenomenon is particularly concerning when some clients use very low ranks (e.g., rank 1 or 4) to accommodate strict resource constraints. When the shared low-rank subspace dominates the aggregated update, higher-rank components may contribute little. The very low rank of the dominant subspace limits its adaptation capacity, which may degrade global model performance. Our experiments across different client rank configurations support this concern, as shown in Figures 2(d), 6(c), and 7(a).

Q1: Definition of accuracy

On p.1, “accuracy” refers to the GSM8K test accuracy of LLaMA-3.1-8B fine-tuned on MetaMathQA40K. On p.7 and in the appendices, it refers to the global model’s test accuracy on the corresponding downstream tasks. We have revised the relevant descriptions and expanded the Metrics paragraph in Section 6 to explicitly define accuracy for each task. These changes are shown in blue on p.8 of the revised manuscript. The added definition is:

“Test accuracy is measured as the percentage of correctly predicted class labels for text and image classification, exact-match accuracy of the final numerical answer for GSM8K, and the percentage of correctly predicted answer choices for commonsense reasoning.”

Q2: Algebraic rank versus effective rank

Algebraic rank is the maximum number of linearly independent rows or columns of a matrix, equivalently the number of strictly nonzero singular values. It counts these singular values without considering their magnitudes. By contrast, effective rank accounts for the distribution of singular values and reflects how strongly the singular-value spectrum is concentrated in a few dominant directions.

Under the rank-collapse phenomenon studied here, the aggregated LoRA update can retain a high algebraic rank while concentrating most of its spectral energy in the top-r1r_1 singular directions. The remaining singular values may be nonzero but much smaller than the top-r1r_1 ones. Thus, the additional directions remain present algebraically but contribute little to the update.

The concern is therefore that the additional parameters and communication associated with higher client ranks may not translate into meaningful contributions to the global update. The function np.linalg.matrix_rank(matrix, tol=tolerance) may still report a high algebraic rank if the smaller singular values exceed the specified tolerance. However, examining the singular-value spectrum reveals the concentration of energy in the top-r1r_1 directions that a rank count alone can miss.

In closing, we once again thank the reviewer for the highly positive assessment of the overall quality and contributions of our work. We hope that our clarifications have satisfactorily addressed the remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear AC and Reviewers,

We sincerely thank the reviewers and the AC for their time, constructive feedback, and engagement throughout the review process. Their suggestions have greatly helped improve the quality and rigor of our work.

We appreciate the reviewers' recognition of the strengths of our work:

  • Problem identification and theoretical analysis: All three reviewers recognize the value of identifying and theoretically explaining rank collapse. VwpD further considers the theoretical analysis a significant contribution in itself.
  • Methodological effectiveness: All three reviewers acknowledge the effectiveness of raFLoRA, while r1FZ and twrb particularly highlight its simplicity and intuitive design.
  • Empirical validation: The reviewers recognize the extensive experiments and consistent improvements across models and tasks. twrb also highlights the absence of additional client communication cost.
  • Presentation clarity: VwpD praises the exceptional clarity of the paper, including its mathematical analysis.

Reviewer VwpD provided positive feedback on our work. Our responses primarily focused on clarifying ambiguous wording and providing additional explanations in the revised manuscript.

Reviewer twrb confirmed that concerns were addressed and maintained a positive rating.

Reviewer r1FZ confirmed that concerns on Q1&Q2 were addressed.

We also provided further clarifications to address the reviewers' remaining concerns.

W1: For novelty, we clarified that HeteroFL neither identifies nor directly addresses rank collapse in heterogeneous-rank FedLoRA arising from LoRA's low-rank parameterization, unequal contributor coverage, and SVD-based allocation. Our work explains why higher-rank capacity can be progressively underutilized during federated training and introduces rank-partitioned aggregation to preserve higher-rank contributions. We therefore believe that raFLoRA’s contribution goes beyond transferring HeteroFL’s normalization principle to LoRA.

Q3: For rigor and precision, we refined our claims to avoid potential misinterpretation, limiting the observed relationship between higher-rank energy preservation and improved performance to matched data heterogeneity settings. We explained that preserving higher-rank energy may retain more useful information in the global update. Meanwhile, similar energy ratios may yield different accuracies across heterogeneity settings due to increased aggregation difficulties under stronger data heterogeneity.

Additionally, we carefully reviewed the entire main text and made minor wording refinements to improve rigor and precision, with changes highlighted in green.

We again thank the AC and reviewers for their valuable feedback, which has strengthened the theoretical rigor, empirical validation, and overall clarity of our manuscript.

Kind regards,

Authors of Paper #2731

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.