- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FedRespro targets a resource-inverse distribution (RID) scenario in federated low-rank adaptation, where data-rich clients have small LoRA ranks and high-rank clients have less data. It estimates the reliability of complementary high-rank residual directions from low-rank statistical references and cross-round persistence, maintains a server residual memory, and provides rank-matched sketches to clients. Results are reported on Qwen2 and BERT tasks.
Critical strengths
- The paper addresses a practically relevant resource–data mismatch in heterogeneous federated learning, where data availability and model capacity are inversely distributed across clients.
- The proposed framework combines complementary residual extraction, reliability-aware aggregation, and rank-matched knowledge transfer into a coherent design for heterogeneous LoRA collaboration.
- The experiments cover multiple Qwen2 model scales and BERT-based tasks, with comparisons against existing federated LoRA methods and ablation studies evaluating several components.
Critical weaknesses
- The theoretical justification of the reliability estimation and convergence analysis is insufficient. Theorem 1 estimates the uncertainty of complementary residual directions from cosine similarity in the shared subspace, relying on a dimension-ratio relationship between noise energies. However, this relationship does not directly establish that the observed directional discrepancy accurately reflects the uncertainty of complementary residuals, particularly when signal differences and magnitude mismatches are present. Furthermore, the convergence analysis is relatively superficial and lacks a quantitative comparison with existing federated LoRA optimization results in terms of convergence rates, assumptions, and additional error terms.
- The experimental evaluation does not sufficiently validate the proposed SOCD mechanism or its dependence on the RID setting. Although FedRespro achieves promising performance under the constructed RID configurations, the experiments do not directly demonstrate that these improvements arise from recoupling statistical reliability and optimization capacity. In particular, the paper lacks controlled comparisons where clients have equal data sizes, identical LoRA ranks, or non-inversely distributed resources. Furthermore, the communication results in Section 4.3 are obtained under a different protocol from the main accuracy experiments, making it difficult to establish the actual accuracy–communication trade-off.
- Figure 1 illustrates the resource asymmetry between clients but does not clearly demonstrate how RID leads to Statistical-Optimization Capability Decoupling (SOCD).
- Several citations in the Introduction are not naturally integrated into the surrounding sentences, affecting readability. The authors should use parenthetical citations (e.g.,
\citep{}) where appropriate and textual citations (e.g.,\citet{}) when the cited authors form part of the sentence.
Questions for authors
- Can the authors rigorously justify the high-probability error guarantee for the practical estimator based on cosine alignment, distinguishing normalized directional discrepancy, magnitude mismatch, signal variation, and complementary noise?
- Can the authors provide controlled experiments with equal data sizes, homogeneous LoRA ranks, and non-inverse resource distributions to clarify whether the performance improvements are specifically attributable to addressing SOCD?
5 comments
Response to Reviewer Qxu7 (2/4): W2 & Q2 — SOCD mechanism and dependence on RID
We thank the reviewer. FedRespro addresses RID through a statistical reference from data-rich, low-rank updates, high-rank complementary-residual assessment using spatial consistency and persistence across rounds, and capacity-matched projection. We examine this mechanism using resource controls, ablations, deployment diagnostics, and communication accounting.
(1) Resource controls
We compare FedRespro and RB-LoRA on BERT-base/DBpedia: 20 clients, 5 participants per round, 15 rounds, 20 local steps per round, the same 3,000 test samples, and three paired seeds.
| Configuration | FedRespro Server (%) | RB-LoRA Server (%) | Gain (pp) |
|---|---|---|---|
| Strong RID (ρ = −0.943) | 41.69 ± 5.08 | 27.80 ± 2.52 | +13.89 |
| Weaker RID (ρ ≈ −0.6) | 38.65 ± 6.38 | 27.60 ± 1.19 | +11.05 |
| Homogeneous ranks, heterogeneous data volumes | 32.81 ± 5.94 | 22.56 ± 5.07 | +10.26 |
| Equal data volumes, homogeneous ranks | 29.61 ± 7.23 | 19.11 ± 4.10 | +10.50 |
Mean ± SD.
The first two controls keep the data partition, rank proportions, and total rank budget identical, changing only the data–rank assignment and hence the Spearman correlation. The server gain changes from 13.89 to 11.05 pp; rank-8 deployment accuracy changes from 51.07% to 46.01% for FedRespro, versus 27.82% and 27.59% for RB-LoRA. Thus, stronger inverse correlation accompanies a larger gain at equal budgets. Under homogeneous ranks, the same-rank fallback retains gains of 10.26 and 10.50 pp with heterogeneous and equal data volumes, respectively, showing optimization benefits under homogeneous conditions.
(2) Component ablations
We include the three-scale ablations from Reviewer 1 W1:
| Configuration | 0.5B | 1.5B | 7B |
|---|---|---|---|
| Full FedRespro | 45.575 | 62.525 | 79.200 |
| Low-rank clients only; no high-rank supplementation | 43.700 | 61.900 | 78.575 |
| Uniform weighting of accumulated memory | 43.250 | 58.650 | 78.075 |
| Current-round memory only; no historical accumulation | 42.900 | 60.400 | 78.150 |
Server accuracy (%).
At 0.5B, 1.5B, and 7B, respectively, high-rank supplementation improves accuracy by 1.875, 0.625, and 0.625 pp within the full fusion procedure. Uniform weighting reduces accuracy by 2.325, 3.875, and 1.125 pp; current-round-only memory reduces it by 2.675, 2.125, and 1.050 pp. Both variants fall below the low-rank-only reference at every scale. These results support the joint roles of supplementation, weighting, and historical retention: accumulation preserves complementary information for subsequent rounds, while weighting affects its contribution. The full method has the highest accuracy in these ablations at all three scales.
(3) Complementary-knowledge diagnostics
On CoLA, holding training settings, client participation, and reliability-score history fixed, replacing accumulated residuals with current-round residuals reduces MCC from 43.85 ± 1.00 to 17.41 ± 5.23. Removing the accumulated complementary component from the final model reduces MCC by 38.25–40.51 points; randomizing its direction while preserving its norm reduces MCC by 39.99–42.85 points. These interventions show that predictions depend on learned complementary directions.
At fixed trainable rank, removing this component reduces rank-2 deployment MCC from 43.26 ± 3.04 to 5.33 ± 1.01, showing that capacity-constrained clients also use accumulated knowledge. Together, the resource controls, ablations, and directional/deployment diagnostics support collaboration through a statistical reference, complementary-knowledge retention, and capacity-matched transfer.
(4) Communication matched to main performance runs
Original Table 4(b)/Appendix H characterize basic adapter exchange using an auxiliary adapter-only dispatch protocol that differs from the main configuration; Appendix C.2 analyzes the residual-sketch channel. We recompute complete communication from the original Table 1 run logs: 40 rounds for Qwen2-1.5B/7B, including initial adapter downloads, adapter uploads, regular low-rank downloads, and residual-sketch downloads. Logical bytes use actual tensor sizes and precision. As in Appendix H, the backbone is preinstalled; serialization, network headers, and evaluation traffic are excluded.
Across three seeds, totals are 1.5329 ± 0.0150 and 3.5498 ± 0.0346 GiB for 1.5B/7B: approximately 63.8% below FLoRA and 66.4% above RB-LoRA at both scales, mainly due to residual-sketch downloads.
Appendix C.2's 1.5× factor counts scalars. BF16 uploads and FP32 downloads yield a byte ratio of approximately in regular rounds and 1.664 cumulatively. We have clarified this distinction and updated Table 4(b)/Appendix H to pair communication and Table 1 accuracy from the same runs.
Response to Reviewer Qxu7 (3/4): W3 & W4 — Figure 1 and citation formatting
3.3 W3 — Figure 1 does not clearly illustrate how RID leads to SOCD
We thank the reviewer for identifying this presentation issue. Figure 1 was intended to motivate the difficulty of knowledge collaboration through the resource differences between two types of clients. Following this suggestion, we have replaced the original Figure 1 to more clearly illustrate the intermediate steps from inverse resource allocation to the separation of capabilities.
The revised figure retains the overall structure and sequentially illustrates: the inverse allocation of data volume and capacity (RID); limited explorable directions at the low-rank end and insufficient statistical support at the high-rank end; the separation of the two capabilities across clients (SOCD); and the resulting challenges of discovering reliable complementary knowledge and absorbing knowledge under capacity constraints. We have revised the figure and caption accordingly, using a conceptual illustration to clarify the connections among RID, SOCD, and the method design.
3.4 W4 — Unnatural citation formatting in the Introduction
We thank the reviewer for the specific suggestion on citation style. We have adjusted the background citations in the Introduction to use \citep{}, making their parenthetical form consistent with the surrounding sentences.
Response to Reviewer Qxu7 (1/4): W1 — Reliability estimation and convergence analysis
We thank the reviewer. The revised reliability and convergence guarantees are conditional, as detailed below.
Under RID, data-rich, low-rank updates provide a shared-space reference for assessing high-rank complementary directions. Theorem 1 and A.3–A.4 distinguish unregularized cosine distance from the implemented regularized, clipped score.
The target is expected complementary noise energy normalized by the population shared reference signal energy. Corollary 1 now gives a conditional moment-based bound that retains covariance calibration, shared-mean mismatch, reference noise/correlation, magnitude mismatch, regularization, and empirical-projector error. Our Q1 response defines all terms and the joint event. The calibration conditions are explicit; the analysis does not assert universal sample-size or inverse-rank decay.
For the reference, let ey=E∣P(GL−EGL)∣F2/∣PEGL∣F, with nonzero denominator. Its normalized noise term is νL=ey2. Better reference reliability can reduce this term at fixed signal scale, but does not remove the other calibration errors.
Following Appendix B.5, for the fixed objective F(X)=∑kpkFk(X), pk≥0, ∑kpk=1, write the effective-state update as
Xt=Wt+Mt,Xt+1=Xt−ηgt,gt=∇F(Xt)+ξt+bt.
Let Ft be the history before round-t sampling/training, including Wt,Mt. Define bt=E[gt∣Ft]−∇F(Xt) and ξt=gt−E[gt∣Ft]. The measurable decomposition bt=bmem,t+brank,t+bloc,t satisfies the bias controls assumed in B.1–B.5:
E∣bt∣F2≤Bt2:=3L2εmem,t2+3L2εrank,t2+3εloc2.
The memory term measures accumulated reliability-estimation error, with εmem,t≤∑τ<tΛτεrel,τ. The rank-projection term is εrank,t2=E[∑kpk∑i>rkσi2(Mt)], the expected spectral energy discarded beyond client capacities. The local term bounds federated optimization drift. Here ∣λτ∣≤Λτ uses deterministic envelopes. B.2 allocates the spatial-score failure budget across participating high-rank clients and assumes a separate temporal-score error bound. With residual norms bounded by GR, the mean-square residual error is bounded by εrel,t2=EQt(δt)+4GR2δt, where Qt is B.2's joint-good-event bound. This includes the failure event before propagating error to accumulated memory.
Assume F is L-smooth and bounded below, E[ξt∣Ft]=0, E[∣ξt∣F2∣Ft]≤σ2, and the above bias control holds. For run length T, using η≤1/(4L) in every round gives
GT:=T1∑t<TE∣∇F(Xt)∣F2≤ηT2(F(X0)−F⋆)+Lησ2+25BˉT2,BˉT2:=T1∑t<TBt2.
As in Appendix B.6, if η=c/T≤1/(4L) is used and variance/error controls hold uniformly over run lengths, then GT≤O(T−1/2)+25BˉT2: