acceptodds
Under review as a conference paper at ICLR 2027

FedRespro: Breaking Resource-Inverse Distribution Dilemma via Federated Residual Projection

Abstract

Heterogeneity has long been a fundamental obstacle to effective model collabora- tion in federated learning (FL). Despite extensive efforts to address data, system, and model heterogeneity, existing methods remain inadequate for a prevalent yet largely overlooked scenario, which we term Resource Inverse Distribution (RID), where data-rich frontline clients are constrained by limited model capacity, while high-capacity clients have access to only limited data. RID disperses statistical reliability and optimization capacity across different clients, inducing Statistical-Optimization Capability Decoupling: low-capacity clients cannot sufficiently ex-plore effective directions beyond their constrained optimization spaces, whereas complementary knowledge discovered by high-capacity clients suffers from sta-tistical uncertainty due to limited observations. To address this dilemma, we pro-pose FedRespro, a federated heterogeneous model collaboration framework that recouples statistical reliability and optimization capacity across heterogeneous clients. Specifically, FedRespro extracts complementary residual knowledge from the expanded optimization spaces of high-capacity clients and suppresses unre-liable information through statistical, spatial, and temporal reliability estimation. It further constructs rank-matched references via capacity-aware low-rank projec-tion, enabling data-rich low-capacity clients to absorb complementary knowledge beyond their local exploration capabilities. By compensating for the statistical unreliability of high-capacity clients and the constrained optimization capabil-ity of data-rich clients, FedRespro enables effective cross-client complementar-ity between statistical and optimization advantages under RID scenarios. Experi-ments show an average server accuracy gain of 2.19 percentage points across three Qwen2 scales over the strongest evaluated baselines. On BERT-base, CoLA MCC (×100) improves by 23.43 points, while DBpedia accuracy improves by 16.81 percentage points, each relative to its strongest evaluated baseline. A reference implementation is available at https://anonymous.4open.science/r/ FedRespro/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Federated Learning, Heterogeneous Model Collaboration, Resource-Inverse Distribution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Qxu7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FedRespro targets a resource-inverse distribution (RID) scenario in federated low-rank adaptation, where data-rich clients have small LoRA ranks and high-rank clients have less data. It estimates the reliability of complementary high-rank residual directions from low-rank statistical references and cross-round persistence, maintains a server residual memory, and provides rank-matched sketches to clients. Results are reported on Qwen2 and BERT tasks.

Critical strengths
  1. The paper addresses a practically relevant resource–data mismatch in heterogeneous federated learning, where data availability and model capacity are inversely distributed across clients.
  2. The proposed framework combines complementary residual extraction, reliability-aware aggregation, and rank-matched knowledge transfer into a coherent design for heterogeneous LoRA collaboration.
  3. The experiments cover multiple Qwen2 model scales and BERT-based tasks, with comparisons against existing federated LoRA methods and ablation studies evaluating several components.
Critical weaknesses
  1. The theoretical justification of the reliability estimation and convergence analysis is insufficient. Theorem 1 estimates the uncertainty of complementary residual directions from cosine similarity in the shared subspace, relying on a dimension-ratio relationship between noise energies. However, this relationship does not directly establish that the observed directional discrepancy accurately reflects the uncertainty of complementary residuals, particularly when signal differences and magnitude mismatches are present. Furthermore, the convergence analysis is relatively superficial and lacks a quantitative comparison with existing federated LoRA optimization results in terms of convergence rates, assumptions, and additional error terms.
  2. The experimental evaluation does not sufficiently validate the proposed SOCD mechanism or its dependence on the RID setting. Although FedRespro achieves promising performance under the constructed RID configurations, the experiments do not directly demonstrate that these improvements arise from recoupling statistical reliability and optimization capacity. In particular, the paper lacks controlled comparisons where clients have equal data sizes, identical LoRA ranks, or non-inversely distributed resources. Furthermore, the communication results in Section 4.3 are obtained under a different protocol from the main accuracy experiments, making it difficult to establish the actual accuracy–communication trade-off.
  3. Figure 1 illustrates the resource asymmetry between clients but does not clearly demonstrate how RID leads to Statistical-Optimization Capability Decoupling (SOCD).
  4. Several citations in the Introduction are not naturally integrated into the surrounding sentences, affecting readability. The authors should use parenthetical citations (e.g., \citep{}) where appropriate and textual citations (e.g., \citet{}) when the cited authors form part of the sentence.
Questions for authors
  1. Can the authors rigorously justify the high-probability error guarantee for the practical estimator based on cosine alignment, distinguishing normalized directional discrepancy, magnitude mismatch, signal variation, and complementary noise?
  2. Can the authors provide controlled experiments with equal data sizes, homogeneous LoRA ranks, and non-inverse resource distributions to clarify whether the performance improvements are specifically attributable to addressing SOCD?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Qxu7 (1/4): W1 — Reliability estimation and convergence analysis

We thank the reviewer. The revised reliability and convergence guarantees are conditional, as detailed below.

  1. Reliability estimation and reference selection.

Under RID, data-rich, low-rank updates provide a shared-space reference for assessing high-rank complementary directions. Theorem 1 and A.3–A.4 distinguish unregularized cosine distance from the implemented regularized, clipped score.

The target is expected complementary noise energy normalized by the population shared reference signal energy. Corollary 1 now gives a conditional moment-based bound that retains covariance calibration, shared-mean mismatch, reference noise/correlation, magnitude mismatch, regularization, and empirical-projector error. Our Q1 response defines all terms and the joint event. The calibration conditions are explicit; the analysis does not assert universal sample-size or inverse-rank decay.

For the reference, let ey=E∣P(GL−EGL)∣F2/∣PEGL∣Fe_y=\sqrt{\mathbb E\\|P(G_{\mathcal L}-\mathbb EG_{\mathcal L})\\|_F^2}/\\|P\mathbb EG_{\mathcal L}\\|_F, with nonzero denominator. Its normalized noise term is νL=ey2\nu_{\mathcal L}=e_y^2. Better reference reliability can reduce this term at fixed signal scale, but does not remove the other calibration errors.

  1. Error propagation and convergence.

Following Appendix B.5, for the fixed objective F(X)=∑kpkFk(X)F(X)=\sum_k p_kF_k(X), pk≥0p_k\ge0, ∑kpk=1\sum_k p_k=1, write the effective-state update as

Xt=Wt+Mt,Xt+1=Xt−ηg^t,g^t=∇F(Xt)+ξt+bt.X_t=W_t+M_t,\qquad X_{t+1}=X_t-\eta\widehat g_t,\qquad\widehat g_t=\nabla F(X_t)+\xi_t+b_t.

Let Ft\mathcal F_t be the history before round-tt sampling/training, including Wt,MtW_t,M_t. Define bt=E[g^t∣Ft]−∇F(Xt)b_t=\mathbb E[\widehat g_t\mid\mathcal F_t]-\nabla F(X_t) and ξt=g^t−E[g^t∣Ft]\xi_t=\widehat g_t-\mathbb E[\widehat g_t\mid\mathcal F_t]. The measurable decomposition bt=bmem,t+brank,t+bloc,tb_t=b_{\mathrm{mem},t}+b_{\mathrm{rank},t}+b_{\mathrm{loc},t} satisfies the bias controls assumed in B.1–B.5:

E∣bt∣F2≤Bt2:=3L2εmem,t2+3L2εrank,t2+3εloc2.\mathbb E\\|b_t\\|_F^2\le B_t^2:=3L^2\varepsilon_{\mathrm{mem},t}^2+3L^2\varepsilon_{\mathrm{rank},t}^2+3\varepsilon_{\mathrm{loc}}^2.

The memory term measures accumulated reliability-estimation error, with εmem,t≤∑τ<tΛτεrel,τ\varepsilon_{\mathrm{mem},t}\le\sum_{\tau < t}\Lambda_\tau\varepsilon_{\mathrm{rel},\tau}. The rank-projection term is εrank,t2=E[∑kpk∑i>rkσi2(Mt)]\varepsilon_{\mathrm{rank},t}^2=\mathbb E[\sum_k p_k\sum_{i > r_k}\sigma_i^2(M_t)], the expected spectral energy discarded beyond client capacities. The local term bounds federated optimization drift. Here ∣λτ∣≤Λτ|\lambda_\tau|\le\Lambda_\tau uses deterministic envelopes. B.2 allocates the spatial-score failure budget across participating high-rank clients and assumes a separate temporal-score error bound. With residual norms bounded by GRG_R, the mean-square residual error is bounded by εrel,t2=EQt(δt)+4GR2δt\varepsilon_{\mathrm{rel},t}^2=\mathbb E Q_t(\delta_t)+4G_R^2\delta_t, where QtQ_t is B.2's joint-good-event bound. This includes the failure event before propagating error to accumulated memory.

Assume FF is LL-smooth and bounded below, E[ξt∣Ft]=0\mathbb E[\xi_t\mid\mathcal F_t]=0, E[∣ξt∣F2∣Ft]≤σ2\mathbb E[\\|\xi_t\\|_F^2\mid\mathcal F_t]\le\sigma^2, and the above bias control holds. For run length TT, using η≤1/(4L)\eta\le1/(4L) in every round gives

GT:=1T∑t<TE∣∇F(Xt)∣F2≤2(F(X0)−F⋆)ηT+Lησ2+52BˉT2,BˉT2:=1T∑t<TBt2.\mathcal G_T:=\frac1T\sum_{t < T}\mathbb E\\|\nabla F(X_t)\\|_F^2\le\frac{2(F(X_0)-F^\star)}{\eta T}+L\eta\sigma^2+\frac52\bar B_T^2,\qquad\bar B_T^2:=\frac1T\sum_{t < T}B_t^2.

As in Appendix B.6, if η=c/T≤1/(4L)\eta=c/\sqrt T\le1/(4L) is used and variance/error controls hold uniformly over run lengths, then GT≤O(T−1/2)+52BˉT2\mathcal G_T\le O(T^{-1/2})+\frac52\bar B_T^2:

  • BˉT2→0\bar B_T^2\to0 gives vanishing average squared gradient norm, with rate depending on bias decay.
  • BˉT2=O(T−1/2)\bar B_T^2=O(T^{-1/2}) gives GT=O(T−1/2)\mathcal G_T=O(T^{-1/2}).
  • lim sup⁡TBˉT2≤B∞2\limsup_T\bar B_T^2\le B_\infty^2 gives lim sup⁡TGT≤52B∞2\limsup_T\mathcal G_T\le\frac52 B_\infty^2.
  1. Comparison of theoretical guarantees.
MethodConditions and metricGuarantee and error sources
FLoRAStrong convexity, smoothness, full participation, bounded gradient moments, unbiased LoRA gradients; expected objective gap.O(1/T)O(1/T); stochastic variance, data heterogeneity, local updates.
FlexLoRALipschitz continuity, bounded parameters, controlled SVD approximation; generalization/sample complexity.Theorem 1 (§3.4) relates sample requirements to SVD approximation error, not an optimization rate over communication rounds TT.
FedResproSmooth, lower-bounded nonconvex objective; conditionally unbiased noise with bounded variance and the bias controls in Appendix B.5; average squared gradient norm.O(T−1/2)+O(BˉT2)O(T^{-1/2})+O(\bar B_T^2); accumulated memory error, rank-projection error, local optimization drift.
AuthorsOpenReview ↗

Response to Reviewer Qxu7 (2/4): W2 & Q2 — SOCD mechanism and dependence on RID

We thank the reviewer. FedRespro addresses RID through a statistical reference from data-rich, low-rank updates, high-rank complementary-residual assessment using spatial consistency and persistence across rounds, and capacity-matched projection. We examine this mechanism using resource controls, ablations, deployment diagnostics, and communication accounting.

(1) Resource controls

We compare FedRespro and RB-LoRA on BERT-base/DBpedia: 20 clients, 5 participants per round, 15 rounds, 20 local steps per round, the same 3,000 test samples, and three paired seeds.

ConfigurationFedRespro Server (%)RB-LoRA Server (%)Gain (pp)
Strong RID (ρ = −0.943)41.69 ± 5.0827.80 ± 2.52+13.89
Weaker RID (ρ ≈ −0.6)38.65 ± 6.3827.60 ± 1.19+11.05
Homogeneous ranks, heterogeneous data volumes32.81 ± 5.9422.56 ± 5.07+10.26
Equal data volumes, homogeneous ranks29.61 ± 7.2319.11 ± 4.10+10.50

Mean ± SD.

The first two controls keep the data partition, rank proportions, and total rank budget identical, changing only the data–rank assignment and hence the Spearman correlation. The server gain changes from 13.89 to 11.05 pp; rank-8 deployment accuracy changes from 51.07% to 46.01% for FedRespro, versus 27.82% and 27.59% for RB-LoRA. Thus, stronger inverse correlation accompanies a larger gain at equal budgets. Under homogeneous ranks, the same-rank fallback retains gains of 10.26 and 10.50 pp with heterogeneous and equal data volumes, respectively, showing optimization benefits under homogeneous conditions.

(2) Component ablations

We include the three-scale ablations from Reviewer 1 W1:

Configuration0.5B1.5B7B
Full FedRespro45.57562.52579.200
Low-rank clients only; no high-rank supplementation43.70061.90078.575
Uniform weighting of accumulated memory43.25058.65078.075
Current-round memory only; no historical accumulation42.90060.40078.150

Server accuracy (%).

At 0.5B, 1.5B, and 7B, respectively, high-rank supplementation improves accuracy by 1.875, 0.625, and 0.625 pp within the full fusion procedure. Uniform weighting reduces accuracy by 2.325, 3.875, and 1.125 pp; current-round-only memory reduces it by 2.675, 2.125, and 1.050 pp. Both variants fall below the low-rank-only reference at every scale. These results support the joint roles of supplementation, weighting, and historical retention: accumulation preserves complementary information for subsequent rounds, while weighting affects its contribution. The full method has the highest accuracy in these ablations at all three scales.

(3) Complementary-knowledge diagnostics

On CoLA, holding training settings, client participation, and reliability-score history fixed, replacing accumulated residuals with current-round residuals reduces MCC from 43.85 ± 1.00 to 17.41 ± 5.23. Removing the accumulated complementary component from the final model reduces MCC by 38.25–40.51 points; randomizing its direction while preserving its norm reduces MCC by 39.99–42.85 points. These interventions show that predictions depend on learned complementary directions.

At fixed trainable rank, removing this component reduces rank-2 deployment MCC from 43.26 ± 3.04 to 5.33 ± 1.01, showing that capacity-constrained clients also use accumulated knowledge. Together, the resource controls, ablations, and directional/deployment diagnostics support collaboration through a statistical reference, complementary-knowledge retention, and capacity-matched transfer.

(4) Communication matched to main performance runs

Original Table 4(b)/Appendix H characterize basic adapter exchange using an auxiliary adapter-only dispatch protocol that differs from the main configuration; Appendix C.2 analyzes the residual-sketch channel. We recompute complete communication from the original Table 1 run logs: 40 rounds for Qwen2-1.5B/7B, including initial adapter downloads, adapter uploads, regular low-rank downloads, and residual-sketch downloads. Logical bytes use actual tensor sizes and precision. As in Appendix H, the backbone is preinstalled; serialization, network headers, and evaluation traffic are excluded.

Across three seeds, totals are 1.5329 ± 0.0150 and 3.5498 ± 0.0346 GiB for 1.5B/7B: approximately 63.8% below FLoRA and 66.4% above RB-LoRA at both scales, mainly due to residual-sketch downloads.

Appendix C.2's 1.5× factor counts scalars. BF16 uploads and FP32 downloads yield a byte ratio of approximately 5/35/3 in regular rounds and 1.664 cumulatively. We have clarified this distinction and updated Table 4(b)/Appendix H to pair communication and Table 1 accuracy from the same runs.

AuthorsOpenReview ↗

Response to Reviewer Qxu7 (3/4): W3 & W4 — Figure 1 and citation formatting

3.3 W3 — Figure 1 does not clearly illustrate how RID leads to SOCD

We thank the reviewer for identifying this presentation issue. Figure 1 was intended to motivate the difficulty of knowledge collaboration through the resource differences between two types of clients. Following this suggestion, we have replaced the original Figure 1 to more clearly illustrate the intermediate steps from inverse resource allocation to the separation of capabilities.

The revised figure retains the overall structure and sequentially illustrates: the inverse allocation of data volume and capacity (RID); limited explorable directions at the low-rank end and insufficient statistical support at the high-rank end; the separation of the two capabilities across clients (SOCD); and the resulting challenges of discovering reliable complementary knowledge and absorbing knowledge under capacity constraints. We have revised the figure and caption accordingly, using a conceptual illustration to clarify the connections among RID, SOCD, and the method design.

3.4 W4 — Unnatural citation formatting in the Introduction

We thank the reviewer for the specific suggestion on citation style. We have adjusted the background citations in the Introduction to use \citep{}, making their parenthetical form consistent with the surrounding sentences.

Reviewer_mpoXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the problem of Statistical‑Optimization Capability Decoupling (SOCD) in Resource‑Inverse Distribution (RID), a federated‑learning heterogeneity where data‑rich clients have limited model capacity while high‑capacity clients own scarce data. It proposes a LoRA‑based framework named FedRespro. It exploits spatial consistency and cross‑round persistence to filter noisy complementary residuals from high‑capacity clients, accumulates reliable knowledge at server, and uses rank‑matched projection to transfer knowledge to low‑capacity clients. Experiments on Qwen2 and BERT‑base show performance improvements, with ablations, overhead analysis and open‑sourced code.

Critical strengths
  1. It uses spatial consistency and cross‑round persistence to filter noisy updates from data‑poor high‑capacity clients, with formal uncertainty estimator and error bound.

  2. Ablations confirm that cumulative residual memory and reliability weighting contribute meaningfully to performance.

Critical weaknesses
  1. The inverse correlation between local‑data volume and LoRA rank is manually enforced. There are no experiments on partial/noisy RID scenarios or real‑world client traces with naturally‑occurring resource inverse distribution.

  2. The complementary‑residual weight λ needs careful re‑tuning for different model sizes. No adaptive scheduling strategy is provided.

  3. Missing comparisons against non‑LoRA heterogeneous FL methods.

Questions for authors
  1. Compared with other heterogeneous FL baselines, such as federated knowledge distillation methods, what is the unique advantage of your residual‑projection mechanism specifically under RID scenario?

  2. Could you discuss how FedRespro would adapt to non‑LoRA full‑model heterogeneous federated learning settings?

  3. Could you discuss how FedRespro would behave under client drop‑out scenarios? Are there any risks that residual memory accumulates biased signals under unstable client participation?

  4. What is the server‑side memory overhead when scaling to many LoRA layers and very large number of clients?

7 comments
AuthorsOpenReview ↗

Response to Reviewer mpoX (1/6): W1 — Construction of RID and its real-world relevance

We thank the reviewer for the positive assessment and constructive suggestions. We have added analyses and experiments on resource distributions, adaptive fusion, comparisons across federated learning paradigms, client dropout, and server overhead. We address these points below.

We thank the reviewer for raising the question of how representative the RID setting is.

The purpose of constructing RID is to examine, under controlled conditions, how a mismatch between statistical support and adaptation capacity affects knowledge collaboration across clients. In the experimental settings of Section 4 and Appendices D.1/F.1, we inversely allocate data volume and LoRA rank, such that data-rich clients are constrained by lower adaptation capacity, while high-rank clients learn from less data. This design highlights the separation of these two capabilities across clients and provides explicit conditions for analyzing collaboration between shared and complementary knowledge.

Nevertheless, variations in resource correlations beyond the controlled setting, as well as real-device scenarios, also warrant investigation. The reviewer's question motivates further examination of the method's adaptability to different resource distributions. We therefore add controlled experiments with weaker RID and homogeneous client configurations, as well as a real-device telemetry task.

FedRespro retains its performance advantage under both weaker RID and homogeneous configurations. The DBpedia results are shown below.

Experimental conditionFedRespro (%)RB-LoRA (%)Gain (pp)
Strong RID (ρ = −0.943)41.69 ± 5.0827.80 ± 2.52+13.89
Weaker RID (ρ ≈ −0.603)38.65 ± 6.3827.60 ± 1.19+11.05
Homogeneous ranks, heterogeneous data volumes32.81 ± 5.9422.56 ± 5.07+10.26
Equal data volumes, homogeneous ranks29.61 ± 7.2319.11 ± 4.10+10.50

Server accuracy is reported as mean ± standard deviation across three random seeds.

These results indicate that FedRespro's gains do not depend on a strict inverse correlation between resources. When the negative correlation weakens from −0.943 to approximately −0.603, the advantage over RB-LoRA remains 11.05 pp. After removing rank heterogeneity, and even after further eliminating differences in data volume, the improvement remains approximately 10 pp. We have added this DBpedia resource-configuration comparison and its experimental settings to Appendix F.2.

The GreenHub experiments extend the evaluation to a real-device telemetry task. We evaluate a historical implementation of FedRespro on an MLP classification task using 71-dimensional telemetry features. Training ranks are determined by a mapping from device-resource proxies, and we compare natural and shuffled resource mappings.

FedRespro achieves higher mean performance under both natural and shuffled mappings. Under the natural mapping with independent hyperparameter tuning, server balanced accuracy is 69.98% for FedRespro and 69.39% for RB-LoRA, while client deployment balanced accuracy is 69.95% and 69.48%, respectively. Under the shuffled mapping, server balanced accuracy is 69.91% and 69.69%, respectively. These results provide additional task evidence beyond language models.

AuthorsOpenReview ↗

Response to Reviewer mpoX (2/6): W2 — Model-scale sensitivity and adaptive λ

We thank the reviewer for highlighting model-scale sensitivity and suggesting adaptation. λ controls complementary-residual fusion and should reflect training state and residual reliability. The original experiments use fixed weights. We now explore a reliability-based heuristic and calibration-loss-driven adjustment, using aggregation statistics and deployment feedback, respectively.

(1) Reliability-based heuristic

Let st−1,pt−1s_{t-1},p_{t-1} be the previous round's mean spatial and temporal reliability scores of high-rank clients, and at−1=∣GLt−1∣Fa_{t-1}=\\|G_L^{t-1}\\|_F, ct−1=∣Rrelt−1∣Fc_{t-1}=\\|R_{\mathrm{rel}}^{t-1}\\|_F the weighted low-rank update and reliability-weighted residual magnitudes. With valid statistics,

u_t=0.5+1.5q_{t-1}\frac{a_{t-1}}{a_{t-1}+c_{t-1}+10^{-12}},$$ $$\lambda_t=\mathrm{clip}_{[0.5,2]}(0.8\lambda_{t-1}+0.2u_t),\quad\lambda_0=1.$$ Holding other inputs fixed, higher reliability raises the target weight, while a larger relative residual lowers it; smoothing reduces fluctuations. This reuses aggregation statistics without calibration loss or candidate evaluation. We retain the previous λ if a required rank group or valid statistics are unavailable, or non-finite values occur. **(2) Online calibration** Qwen2-0.5B and Qwen2-1.5B each use three seeds and five rounds. All six runs share $\lambda_0=1$, $\epsilon=0.1$, $\eta=0.2$ and one update per round: $$\lambda_t^-=\max(0,\lambda_{t-1}-\epsilon),\quad \lambda_t^+=\min(2,\lambda_{t-1}+\epsilon),$$ $$\widehat g_t=\frac{\mathcal L_{\mathrm{cal}}(\lambda_t^+)-\mathcal L_{\mathrm{cal}}(\lambda_t^-)}{\lambda_t^+-\lambda_t^-},\quad \lambda_t=\mathrm{clip}_{[0,2]}[\lambda_{t-1}+\mathrm{clip}_{[-0.2,0.2]}(-\eta\widehat g_t)].$$ Both candidates follow the actual aggregation and projection-for-dispatch procedures and use the same calibration samples. We then restore the original state and aggregate with the updated λ. Thus λ stays in [0,2], changes by at most 0.2 per round, and receives five updates per run, with **the same controller across model scales**. We split 6,000 samples from the Commonsense170K training source into 5,398 training, 301 calibration, and 301 independent model-selection samples. The subsets are disjoint; samples whose instruction exactly matches a test instruction are excluded. Each round, one client from each of R4 and R8 is sampled among all 40 clients, potentially beyond the six training participants. Each uses the first four samples of its fixed calibration set; both candidates share these eight samples. After projection for dispatch, we compute answer-token cross-entropy, exclude prompt tokens, weight losses within each rank group by answer-token counts, and average the two group losses equally. **Accuracy (%, mean ± sample SD)** | Model | λ strategy | Server | R4 | R8 | | --- | --- | --- | --- | --- | | Qwen2-0.5B | Fixed λ=1 | 41.24±3.63 | 40.02±3.53 | 39.14±3.84 | | | Heuristic | 39.37±2.08 | 38.99±1.07 | 37.61±2.84 | | | Learnable | 40.63±2.56 | 36.20±0.60 | 36.72±0.78 | | Qwen2-1.5B | Fixed λ=2 | 63.23±1.18 | 61.74±1.29 | 61.50±1.21 | | | Heuristic | 59.90±0.93 | 63.08±0.74 | 62.52±0.41 | | | Learnable | 62.84±1.60 | 60.54±1.48 | 60.91±1.23 | Fixed weights are references from the original main configurations; online calibration uses five-round supplementary runs. **This is not an equal-training-budget comparison.** Learnable server means differ from the fixed references by 0.61 and 0.39 pp; the table also reports both deployment views. **Online calibration overhead (three seeds, five rounds)** | Model | Training comm. (MiB) | Extra cal. comm. (MiB) | Increase | Extra cal. time (s) | | --- | --- | --- | --- | --- | | Qwen2-0.5B | 120.14 | 92.81 | ≈77.3% | 125.6±45.4 | | Qwen2-1.5B | 242.10 | 187.03 | ≈77.3% | 255.8±25.3 | Each round adds two candidate aggregations, four candidate downloads, and two evaluations of eight samples (16 sample-level forward computations). Clients return losses and counts without gradients. Communication increases are extra calibration bytes divided by regular training bytes per seed, then averaged. These experiments demonstrate feasible online fusion adjustment, including deployment-feedback-based adjustment with one controller across two model scales. We have added the strategies, settings, results, and overhead analyses to Appendix I.2.
AuthorsOpenReview ↗

Response to Reviewer mpoX (3/6): Q1 (combined with W3) — Non-LoRA comparisons and advantages under RID

We thank the reviewer for suggesting broader comparisons and clarification of FedRespro's mechanism under RID.

The original baselines maintain a common fine-tuning framework to focus on heterogeneous-update aggregation. FedRespro uses a shared frozen backbone with different-rank adapters; HetLoRA, FlexLoRA, FLoRA, RB-LoRA, and FSLoRA therefore compare aggregation of the same type of updates.

The reviewer's suggestion extends this comparison to other knowledge-fusion paradigms. HeteroFL uses different-sized subnetworks, FedRolex uses rolling submodels, and FedDF uses ensemble outputs and server-side distillation. These mechanisms differ in trainable parameters, transmitted content, and server computation. We therefore add experiments based on FedDF's ensemble-distillation mechanism to examine both communication requirements and predictive performance, and relate residual projection to RID.

E14 uses BERT/DBpedia with the same 20 clients, data partition, and 5 participating clients per round. FedRespro uses ranks 2/4/8; FedDF trains different-depth BERT models with all parameters trainable and uses 64 unlabeled proxy samples for distillation. Two depth configurations (2/4/6 and 4/8/12 layers) and two schedules (5 rounds × 5 steps and 1 round × 25 steps) yield four FedDF candidates.

None can complete its first-round parameter exchange within the five-round FedRespro budget of 65.41 MiB. The first complete exchanges for the two FedDF depth configurations require 1.861 and 2.812 GiB, approximately 29.1× and 44.0× that budget. Raising the cap to 256× allows all four candidates to complete their prescribed training:

Method and configurationTotal communicationRelative to FedRespro
FedRespro (ranks 2/4/8, 5 rounds)65.41 MiB1.0×
FedDF (2/4/6 layers, 1 round × 25 steps)1.861 GiBApprox. 29.1×
FedDF (4/8/12 layers, 1 round × 25 steps)2.812 GiBApprox. 44.0×
FedDF (2/4/6 layers, 5 rounds × 5 steps)9.939 GiBApprox. 155.6×
FedDF (4/8/12 layers, 5 rounds × 5 steps)15.325 GiBApprox. 239.9×

Using the independent selection set, we selected FedDF with 4/8/12 layers and 5 rounds. Its test accuracies and FedRespro's deployment results follow; capacity groups denote LoRA ranks for FedRespro and BERT depths for FedDF.

MethodLow capacityMedium capacityHigh capacity
FedResprorank 2: 7.83%rank 4: 7.83%rank 8: 8.17%
FedDF4 layers: 10.17%8 layers: 16.17%12 layers: 12.67%

The selected FedDF configuration achieves higher test accuracy in all three groups while using approximately 240× the cumulative communication. This illustrates the different communication requirements of full-model exchange and heterogeneous LoRA fine-tuning, and their performance–communication relationship under these configurations.

FedRespro's RID-specific design connects a statistical reference, complementary-direction assessment, and capacity-matched transfer. In the alignable update space of a common backbone, it constructs a shared reference from data-rich, low-rank updates, separates complementary residuals from high-rank updates, and adjusts their retention and fusion through spatial consistency and persistence across rounds. Capacity-matched projection then transfers the fused information to clients of different ranks. These stages address statistical support, complementary-direction reliability, and information use under capacity constraints, forming a knowledge-collaboration procedure for SOCD.

FedRespro and FedDF fuse knowledge in parameter-update and prediction-output spaces, respectively. FedDF transfers ensemble knowledge on proxy samples and supports broader architectural heterogeneity. FedRespro uses common-backbone alignment to explicitly handle shared and complementary components without proxy-data distillation during fusion.

Reviewer_ykLYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of heterogeneous federated fine-tuning under a "Resource Inverse Distribution" (RID) setting. In this scenario, clients with abundant local data are assigned lower model capacities, while clients with higher model capacities possess only a limited number of local samples. To address the resulting mismatch between statistical reliability and optimization capability, the authors propose the FedRespro framework. Experiments conducted using Qwen2 models of various scales (as well as a BERT-base model) demonstrate that this approach outperforms several baseline methods based on heterogeneous LoRA.

Critical strengths
  1. The overall design of the method is well-conceived. Specifically, it treats data-rich, low-capacity clients as sources of statistical reference, while extracting complementary optimization directions from high-capacity clients; it then facilitates cross-capacity knowledge transfer via residual memory and low-rank projection.
  2. The experiments cover a range of model scales, including Qwen2-0.5B, 1.5B, 7B, and BERT-base. FedRespro achieves the highest server-side accuracy across all three Qwen2 model scales.
Critical weaknesses
  1. Is the performance gain primarily derived from the additional server-side residual memory?
  2. On the CoLA dataset, FedRespro achieves a score of 43.00, whereas FlexLoRA, FLoRA, RB-LoRA, and FSLoRA score only 6.17, 0.60, 19.57, and 11.55, respectively; the authors do not provide a sufficient explanation for this significant performance gap.
  3. To ensure a fair comparison, the authors should report communication efficiency using identical configurations for FedRespro.
  4. The reliability estimation for complementary residuals relies on relatively strong theoretical assumptions.
Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ykLY (1/4): W1 — Are the performance gains primarily attributable to additional server-side residual memory?

We thank the reviewer for the careful review. We understand that the reviewer raises four concerns: the contribution of server-side residual memory, the substantial performance gap on CoLA, the fairness of the communication-efficiency comparison, and the theoretical assumptions underlying reliability estimation. We address each concern below.

Reviewer comment (verbatim)

Is the performance gain primarily derived from the additional server-side residual memory?

Response

We thank the reviewer for this important question. We agree that server-side residual memory is an important component of FedRespro and that further ablations are needed to distinguish the role of historical accumulation from that of memory utilization. To address this question, we compare the full method with three simplified configurations at three model scales, 0.5B, 1.5B, and 7B: a reference using only low-rank clients without high-rank supplementation, uniform weighting of accumulated memory, and current-round memory without historical accumulation. The server accuracies are reported below.

Configuration0.5B1.5B7B
Full FedRespro45.57562.52579.200
Low-rank clients only, without high-rank supplementation43.70061.90078.575
Uniform weighting of accumulated memory43.25058.65078.075
Current-round memory only, without historical accumulation42.90060.40078.150

First, historical accumulation does contribute to the performance of the full method. Using only current-round memory without historical accumulation reduces accuracy by 2.675, 2.125, and 1.050 pp for the three models, respectively. This result supports the effectiveness of retaining complementary information across rounds.

Second, retaining historical memory alone is insufficient to achieve the performance of the full method; memory weighting also matters. When historical accumulation is retained but the original memory weighting is replaced with uniform weighting, accuracy decreases by 2.325, 3.875, and 1.125 pp, respectively. This indicates that, beyond retaining historical memory, how it is weighted and utilized also affects performance.

Finally, the benefit of high-rank supplementation should be interpreted in conjunction with how it is utilized. Compared with the reference using only low-rank clients without high-rank supplementation, full FedRespro improves accuracy by 1.875, 0.625, and 0.625 pp, respectively. Meanwhile, both simplified variants, uniform weighting and current-round memory only, perform below the low-rank-only reference at all three model scales.

These ablations therefore support a more precise explanation of the gains: historical accumulation provides valuable information across rounds, memory weighting further affects how effectively this information is utilized, and the full configuration achieves the highest accuracy at all three model scales. We have added these ablation results to Appendix G.3 and clarified the experimental support for server-side residual memory and its weighted utilization.

AuthorsOpenReview ↗

Response to Reviewer ykLY (2/4): W2 — The substantial performance gap on CoLA

Reviewer comment (verbatim)

On the CoLA dataset, FedRespro achieves a score of 43.00, whereas FlexLoRA, FLoRA, RB-LoRA, and FSLoRA score only 6.17, 0.60, 19.57, and 11.55, respectively; the authors do not provide a sufficient explanation for this significant performance gap.

Response

We thank the reviewer for this important question. To understand the substantial performance gap on CoLA, we further examined the data distribution, the prediction behavior of the baselines, and the role of complementary knowledge retention in FedRespro. We present the analysis below.

First, the inverse allocation of data volume and model capacity in the CoLA experiments makes learning more challenging. Under the RID configuration in Section 4 and Appendices D.1 and F.1, clients with lower LoRA ranks receive more training data. Specifically, rank-2 clients have an average of 563.7 samples, whereas rank-8 clients have only 230.3 samples on average. Thus, clients with stronger statistical support are constrained by their adaptation capacity, while clients with greater capacity must learn from fewer samples. Together with the non-IID data partition (Dirichlet α = 0.5) and partial participation (5 out of 20 clients sampled per round), this setting makes it more difficult to exploit complementary knowledge while avoiding unreliable updates.

Second, our additional analysis identifies a common failure mode across the baselines: predictions are heavily biased toward the majority class. The majority class accounts for 69.13% of the CoLA validation set. However, all 12 final parameter-averaged models across the four baselines and three random seeds predict the majority class for more than 95% of the samples. The average minority-class recall is only 0.41%, 0.10%, 8.07%, and 3.62% for FlexLoRA, FLoRA, RB-LoRA, and FSLoRA, respectively. In particular, two FLoRA runs predict the majority class for every sample, yielding an accuracy of 69.13% but an MCC of zero. These results indicate that the low baseline scores are closely associated with difficulty distinguishing the two classes.

Third, FedRespro alleviates this issue by identifying and retaining complementary knowledge across communication rounds. Rather than directly merging heterogeneous updates, FedRespro uses updates from data-rich clients to construct a shared reference and identifies additional update directions contributed by high-rank clients. It then assesses the reliability of these directions using their consistency with the shared knowledge and accumulated historical information. By retaining useful complementary directions across rounds, information contributed by intermittently participating clients can continue to influence subsequent optimization.

The ablations provide direct evidence for the importance of this mechanism. In the original experiments, replacing residual accumulation across rounds with current-round residuals alone reduces CoLA MCC from 43.00 ± 0.83 to 15.35 ± 4.13. The additional controlled experiments yield consistent results while keeping the training configuration, client participation, and reliability-score history unchanged (43.85 ± 1.00 versus 17.41 ± 5.23). Moreover, removing the accumulated complementary component from the final model reduces MCC by 38.25–40.51 points; randomizing its direction while preserving its norm produces a reduction of similar magnitude (39.99–42.85 points). These results indicate that the gains depend on the learned complementary directions, rather than merely on the presence or magnitude of additional updates.

This knowledge also benefits capacity-constrained clients. With the trainable rank held fixed, removing the accumulated complementary component reduces the MCC of rank-2 client models from 43.26 ± 3.04 to 5.33 ± 1.01.

Taken together, the substantial performance gap on CoLA can be understood from two perspectives: the baselines exhibit difficulty learning class-discriminative representations under RID, whereas FedRespro can retain useful complementary information across rounds.

AuthorsOpenReview ↗

Response to Reviewer ykLY (3/4): W3 — Comparing communication efficiency under identical configurations

We thank the reviewer. We agree that communication costs should correspond to the main performance experiments and include all model-parameter transmissions. We therefore recomputed them from the logs of the original Table 1 runs and clarify the original table's scope and the relation between Appendix C.2 and actual byte counts.

(1) Purpose and scope of the original analysis

In the original submission, Table 4(b) and Appendix H used an auxiliary adapter-only dispatch protocol to characterize basic adapter-exchange costs, while Appendix C.2 analyzed the additional residual-sketch channel. These analyses separated the costs of adapter exchange and complementary-information transfer.

The auxiliary protocol differs from the main performance configuration, so its costs cannot be paired directly with Table 1 accuracies. Although it helps explain the communication mechanism, evaluating the complete method requires accounting for all model tensors transmitted in the main runs.

(2) Complete accounting from the original main performance logs

The following results are recomputed from the original Table 1 logs for FedRespro, FLoRA, and RB-LoRA: 40 rounds for Qwen2-1.5B and Qwen2-7B, with each method's original model configuration, participation, ranks, and rounds. Thus, costs and accuracies correspond at the run level.

We count logical tensor bytes using actual tensor sizes and numerical precision, including initial adapter downloads, adapter uploads, regular low-rank downloads, and FedRespro's residual-sketch downloads. Dense residual memory retained on the server incurs no transmission cost itself; sketches sent to clients are counted. Following Appendix H, the backbone is preinstalled; serialization, network headers, and evaluation traffic are excluded. We accumulate costs per seed and report mean ± sample standard deviation across three seeds.

ModelMethodUplinkDownlinkTotal
Qwen2-1.5BFLoRA0.3098 ± 0.00313.9303 ± 0.02154.2401 ± 0.0245
RB-LoRA0.3098 ± 0.00310.6115 ± 0.00590.9214 ± 0.0090
FedRespro0.3098 ± 0.00311.2231 ± 0.01191.5329 ± 0.0150
Qwen2-7BFLoRA0.7175 ± 0.00729.1018 ± 0.04989.8193 ± 0.0567
RB-LoRA0.7175 ± 0.00721.4162 ± 0.01382.1337 ± 0.0209
FedRespro0.7175 ± 0.00722.8324 ± 0.02753.5498 ± 0.0346

Cumulative communication over 40 rounds, in GiB; mean ± sample standard deviation across three seeds.

At both scales, FedRespro uses approximately 63.8% less communication than FLoRA and 66.4% more than RB-LoRA. FedRespro and RB-LoRA have identical uplink costs; the additional cost primarily comes from residual-sketch downloads.

(3) Reconciling the 1.5× scalar ratio with the 1.664× byte ratio

Appendix C.2, Eqs. (109)–(112), counts transmitted scalars. Let QQ be the number of scalars needed for low-rank representations matched to participating clients' ranks in one round. Standard communication uploads and downloads QQ scalars each. When sketch rank equals client rank, sk=rks_k=r_k, FedRespro adds a residual download of QQ scalars. Ignoring small metadata costs, the ratio is

2Q+Q2Q=1.5.\frac{2Q+Q}{2Q}=1.5.

This is also the byte ratio only if all channels use the same precision. The main runs use BF16 adapter uploads and FP32 downloads, including FP32 residual sketches. Regular-round byte costs are therefore 2Q+4Q2Q+4Q for standard low-rank communication and 2Q+4Q+4Q2Q+4Q+4Q for FedRespro:

2Q+4Q+4Q2Q+4Q=53≈1.667.\frac{2Q+4Q+4Q}{2Q+4Q}=\frac{5}{3}\approx1.667.

Thus, regular-round scalar counts increase by approximately 50%, while byte costs increase by approximately two-thirds. Actual cumulative records yield a ratio of approximately 1.664 for both models, or 66.4% more than RB-LoRA.

We have updated Section 4.3, Table 4(b), and Appendix H with this complete accounting and distinguished scalar counts from actual-precision byte counts in the main text and Appendix C.2. This avoids interpreting the 1.5× scalar ratio as the main configuration's byte-cost ratio.

AuthorsOpenReview ↗

Response to Reviewer ykLY (4/4): W4 and Questions for Authors — Theoretical assumptions

1.4 W4 — Strong theoretical assumptions underlying reliability estimation

Reviewer comment (verbatim)

The reliability estimation for complementary residuals relies on relatively strong theoretical assumptions.

Response

We thank the reviewer for this comment. Explicit assumptions are necessary to formalize the RID setting and characterize the optimization relationships among client models involved in SOCD. Assumptions 1–5 in Appendix A.2 draw on existing frameworks in heterogeneous federated learning and related theoretical analyses, with their specific forms reflecting the structural characteristics of RID.

Assumptions 1–2 specify aligned and nested update spaces and the decomposition into shared and client-specific signals. Assumption 3 bounds finite-sample estimation error, Assumption 4 ensures the stability of the reference subspace through spectral separation, and Assumption 5 controls anisotropic noise, with an additional sub-Gaussian condition for the high-probability analysis. These conditions serve explicit mathematical purposes. For example, for zero-mean vectorized noise ξk\xi_k with covariance Σk\Sigma_k and a fixed orthogonal projector PP, if mI⪯Σk⪯MImI\preceq\Sigma_k\preceq MI, then

\leq \mathbb{E}\\|P\xi_k\\|_2^2 =\operatorname{tr}(P\Sigma_k) \leq M\\,\operatorname{rank}(P).$$ Thus, the covariance bounds provide quantitative control for comparing projected noise energy through subspace dimensions, while allowing different clients to have different anisotropic noise distributions. Similar structural conditions also appear in federated LoRA theory. [FLoRA (Wang et al., 2024, Appendix B)](https://proceedings.neurips.cc/paper_files/paper/2024/file/28312c9491d60ed0c77f7fff4ad86dd1-Paper-Conference.pdf) assumes smoothness, strong convexity, bounded stochastic-gradient moments, and unbiased LoRA updates in its convergence analysis; [FlexLoRA (Bai et al., 2024, Section 3.4)](https://papers.neurips.cc/paper_files/paper/2024/file/1a134b50202088aa8c595cc99b310e5a-Paper-Conference.pdf) assumes Lipschitz continuity, bounded parameters, and bounded SVD approximation error in its generalization analysis. These studies provide a basis for adopting explicit structural and error-control conditions suited to the theoretical question. FedRespro follows this analytical approach to study shared and complementary subspaces under RID while retaining heterogeneity in client data distributions, model ranks, and noise covariances. Strict nesting and spectral stability delineate the setting covered by the theoretical guarantees and support a conditional analysis of SOCD; they do not extend the claims regarding FedRespro's applicability or effectiveness beyond that setting. ### 1.5 Questions for Authors The reviewer refers to “See Weaknesses.” The corresponding responses are provided in W1–W4 above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.