- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the rank collapse phenomenon in heterogeneous rank FedLoRA, pointing out that uniform aggregation weights weaken high-rank components that are only supported by a subset of clients. The authors analyze the cumulative process of this attenuation under simplified assumptions and propose raFLoRA, which employs normalization based on the clients effectively contributing to the components. Experiments across vision, language, and reasoning tasks validate the method's effectiveness.
Critical strengths
-
The problem analysis is valuable. The paper links high-rank energy decay with client coverage, providing a more explanatory analysis than simply comparing accuracy.
-
The method is simple and is well supported by experiments. The results demonstrate the practical application of high-rank components without increasing client communication volume.
Critical weaknesses
-
The conclusions regarding geometric decay rely on strong assumptions, such as fixed singular directions and the absence of singular value crossings. The appendix discusses basis changes, but it has not yet proven that rank collapse still occurs under perturbations, nor has it quantified the degree of deviation of actual training from these assumptions. Therefore, the theory's explanation of real training remains limited.
-
The gains on some tasks are modest. For example, the average accuracy of commonsense reasoning of the 8B model is only 0.27 percentage points higher than that of FlexLoRA.
Questions for authors
-
Could you provide data on the evolution of singular subspaces or instances of singular value crossing during actual training to illustrate the applicability of the stable spectral routing hypothesis?
-
For inference tasks where performance gains are modest, are the overall inference benefits stable?
4 comments
2) One-step fit of the higher-rank energy fraction. We define the observed direction energy as and the higher-rank energy fraction as . For a client supporting direction , we measure . Let and be its empirical first and second moments, and let . Under the local fixed-subspace approximation, the expected one-step contraction factor is , where . This gives and .
Over the valid adjacent-round transitions, we report , , and , where is the mean of the observed fractions over the valid transitions. Lower and indicate smaller fitting errors, while a higher indicates a better fit relative to the mean baseline.
| Partition | Layer | MAE | RMSE | |
|---|---|---|---|---|
| IID | 0 | 0.0196 | 0.0411 | 0.8312 |
| IID | 6 | 0.0184 | 0.0402 | 0.8318 |
| IID | 11 | 0.0061 | 0.0174 | 0.9252 |
| IID | Layer Average | 0.0147 | 0.0329 | 0.8627 |
| non-IID | 0 | 0.0166 | 0.0376 | 0.8101 |
| non-IID | 6 | 0.0127 | 0.0310 | 0.8363 |
| non-IID | 11 | 0.0058 | 0.0147 | 0.8908 |
| non-IID | Layer Average | 0.0117 | 0.0278 | 0.8457 |
The consistently high one-step values show that the approximation captures local energy evolution under both IID and strongly non-IID data. In particular, non-IID partitioning only slightly reduces the averaged from 0.8627 to 0.8457. The deepest Layer 11 yields the best fit in both settings, with values of 0.9252 and 0.8908, respectively. This stronger fit may reflect lower subspace drift and greater concentration of update energy in the dominant singular directions, which could reduce the cross-direction mixing omitted by the stable spectral routing assumption.
W2 & Q2: Magnitude and stability of performance gains
In the main experiments, we report accuracy as mean ± standard deviation across three random seeds. With LLaMA-3.1-8B, raFLoRA shows lower run-to-run variability than FlexLoRA. On GSM8K, the reported variability is ±0.27 for raFLoRA and ±0.74 for FlexLoRA. For the average across eight commonsense reasoning tasks, the corresponding values are ±0.08 for raFLoRA and ±0.67 for FlexLoRA.
Table 11 in Appendix H reports results for each of the eight commonsense reasoning subtasks. raFLoRA outperforms FlexLoRA on five subtasks, with the largest accuracy gains on WG (+1.18), ARC-c (+1.17), and SIQA (+0.95). The results in Sections 6.5 and 6.6, together with the additional experiments in Appendix E, further show that raFLoRA improves performance under stronger data heterogeneity and across a range of client rank configurations and distributions.
We have added the experimental results described above to the revised manuscript, with the new text shown in dark purple.
In closing, we would like to once again thank the reviewer for the insightful comments. We hope that our responses have satisfactorily addressed the remaining concerns.
Thank you for the clarification. My concerns have been addressed. I will maintain the original rating.
We sincerely thank the reviewer for maintaining the positive evaluation. We are pleased that our responses have addressed your concerns.
We thank the reviewer for recognizing the value of our analysis and the effectiveness of raFLoRA. To address the reviewer's remaining questions and concerns, we provide the following clarifications.
W1 & Q1: The gap between theory and actual training
Theoretical scope. Appendix C relaxes the stable spectral routing assumption and derives a one-round perturbation bound. Extending this analysis across rounds is challenging. Specifically, singular-value crossings can move directions between partitions with different contributor coverage, while basis drift and mixing can redistribute energy across partitions. These effects couple the partition energies and make their evolution depend on the training trajectory, complicating the derivation of a closed recursion. A multi-round guarantee would require controlling the cumulative perturbations tightly enough for the retention gap to continue driving contraction. Establishing when rank collapse persists under such perturbations is a nontrivial extension that we leave to future work.
The one-round contributor dilution in Eq. (3) follows directly from the aggregation rule and holds independently of stable spectral routing. Because higher-rank partitions have fewer contributors, their aggregated updates may be more sensitive to client participation and alignment. Under non-IID training, weaker alignment may limit the accumulation of higher-rank contributions across rounds, compounding the dilution caused by uniform averaging. In Fig. 5(b), FlexLoRA exhibits the spectral signature of rank collapse under non-IID training. Its higher-rank energy fraction decreases further as data heterogeneity increases.
Empirical assessment. To assess how closely stable spectral routing approximates actual training, we measure singular-subspace overlap between consecutive rounds and compare one-step estimates of the higher-rank energy fraction with observed values. The shared top-8 subspace has a mean overlap of 98.07% under IID training and 97.36% under non-IID training. Across the tracked layers, the one-step model reduces squared error relative to a baseline that uses each layer’s mean observed fraction by an average of 86.27% under IID training and 84.57% under non-IID training. These results support stable spectral routing as a local approximation for the shared low-rank subspace. The lower overlap for larger subspaces suggests that extending the theory to higher ranks may require accounting for subspace drift.
We evaluate FlexLoRA on CIFAR-100 with ViT-Base, using 100 clients and sampling 10 per round. Client ranks are uniformly distributed over {8,16,32,48,64}. We run 100 rounds under IID and Dirichlet non-IID (α=0.1) partitions with matched seeds and track the query projection updates in layers 0, 6, and 11.
1) Subspace evolution. Let the aggregated update at layer ℓ and round t be ΔWℓ(t)=Uℓ(t)Σℓ(t)(Vℓ(t))⊤. We measure the normalized overlap between consecutive cumulative top-r subspaces as Oℓ,r(t)=2r1(∥(Uℓ,r(t−1))⊤Uℓ,r(t)∥F2+∥(Vℓ,r(t−1))⊤Vℓ,r(t)∥F2). This projector-based metric is invariant to sign changes and rotations within the subspace, and O=1 indicates identical consecutive subspaces. The table reports the mean overlap across the three tracked layers and all 99 adjacent-round transitions.
The shared top-8 subspace has a mean adjacent-round overlap of 98.07% under IID data and 97.36% under non-IID data. Overlap decreases as r increases, and the two settings differ by at most 1.87 percentage points. These results suggest that stable spectral routing is an accurate local approximation for the shared low-rank core, but a weaker approximation for higher-rank directions.