acceptodds
Under review as a conference paper at ICLR 2027

SpaRFL: Sparse Low-Rank Federated Learning for Memory-Constrained Clients

Abstract

Federated Learning (FL) enables collaborative training across decentralized data sources, but standard FL methods typically assume that each participating client can store and optimize the full global model. We study federated Transformer training in the no-client-fits regime, where no participating client can store and train the full dense global model within its local memory budget. To address this setting, we propose SpaRFL (Sparse Low-Rank Federated Learning), a client-specific one-shot sparse+low-rank projection method that constructs trainable client models under heterogeneous memory constraints while preserving a dense server model. Across vision, natural language understanding, and language modeling, SpaRFL consistently outperforms the considered model-heterogeneous baselines under severe resource constraints. Under a projected-linear training-state budget 90% below dense training, SpaRFL remains within 1% of the dense baseline on MRPC. In federated pretraining on C4, LLaMA-130M trained at a 70% reduction in the same budget achieves lower perplexity than the dense LLaMA-60M baseline. Overall, SpaRFL provides a practical approach to federated training under strict client memory constraints.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
federated learning, memory-efficient training, heterogeneous clients, sparse low-rank decomposition, Transformer training, model heterogeneity, resource-constrained learning, model compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kg7tOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors study federated Transformer training in the no-client-fits regime, where no participating client can store and train the full dense global model within its local memory budget. To address this setting, they propose SpaRFL (Sparse Low-Rank Federated Learning), a client-specific one-shot sparse+low-rank projection method that constructs trainable client models under heterogeneous memory constraints while preserving a dense server model.

Critical strengths
  1. The paper is well-written and easy to follow.
  2. The motivation for conducting this work is clear, i.e., to address the no-client-fits regime problem, where no participating client can store and train the full dense global model within its local memory budget.
Critical weaknesses
  1. The novelty of this work is limited. The difference between this work and FedLRS seems to be that the clients have heterogeneous memory budgets, but the method is similar, i.e., training sparse and low-rank components.
  2. The motivation of the proposed method is not clear. In lines 195- 196, "represent each client layer as the sum of a low-rank component and a sparse residual.", why are they split into these two? Could the authors provide theoretical or experimental explanations? Or is it just a follow-up to FedLRS?
  3. The experimental model used for pretraining is too small in size (i.e., only 120M). The authors should conduct experiments on a larger model size, e.g., 7B or larger.
  4. The theory gives little help in choosing the projection. The analysis supports local training in a projected model, but it does not explain when a particular sparse mask or low-rank space keeps the useful Transformer updates, especially when activations or sequence lengths change.
Questions for authors
  1. What are the server costs for projection and rebuilding compared with normal dense aggregation, especially for large language-model layers?
  2. How much client data or activation information is needed for the NLP projection, and can this be computed without new privacy or communication problems?
  3. How sensitive are the results to the sparse/low-rank budget split and to the projection method? When is a fully sparse or fully low-rank method better?
6 comments

We thank the reviewer for the review, and for noting that the paper is well written and that the motivation, training in the no-client-fits regime where no client can store and train the dense global model, is clear. We respond to each point below.

(W1: Novelty relative to FedLRS) We respectfully disagree that the difference to FedLRS reduces to heterogeneous budgets. Sparse+low-rank parameterizations are an established tool (RoSA, LoSparse, SLTrain, FedLRS), and we do not claim the parameterization itself as our contribution. Our contributions are:

  1. Problem formulation. The no-client-fits regime, with client budgets defined as optimizer-aware persistent training state in bytes, separating trainable (14 B) from frozen (2 B) values. FedLRS targets communication and computation cost and assumes a fixed decomposed model on every client.
  2. A dense server with per-round, client-specific, one-shot projection. SpaRFL keeps a dense server model and, in every round, projects it to each client's byte budget. FedLRS instead trains the decomposed components directly and has no dense server model to re-project. The re-projection is what lets a single model serve heterogeneous clients, and what allows a frozen sparse anchor of pretrained weights during fine-tuning.
  3. An activation-space joint sparse+low-rank projection for LLMs. It minimizes the activation-weighted distortion ∥(W−S−L)D∥F2\|(W-S-L)D\|_F^2, with a regime-dependent order (sparse-first for fine-tuning, low-rank-first for pretraining). Prior sparse+low-rank methods operate in weight space, and prior activation-aware methods treat sparsity and low rank separately. The activation-space projection alone improves C4 perplexity by 14.5% over its weight-space counterpart (App. N).
  4. A convergence analysis for factor-space local Adam under both λagg=1\lambda_{\mathrm{agg}}=1 and λagg=0\lambda_{\mathrm{agg}}=0, which identifies the projection residual and the collective tangent coverage as the quantities that matter.

Empirically, these differences are large. At matched budgets on C4, SpaRFL improves over FedLRS by 20.1 PPL on LLaMA-60M (37.94 vs. 58.03) and by 9.4 PPL on LLaMA-130M (31.56 vs. 40.91).

(W2: Why sparse + low-rank?) The rationale has three parts.

  • Structural. Trained weight matrices combine broadly distributed structure, which a few singular directions capture, with a small number of large, localized entries, i.e., outliers, which are expensive to represent at low rank. This is the classical low-rank-plus-sparse model of robust PCA [1, 2], and it has been confirmed for Transformer weights by LoSparse [3] and SLTrain [4]. A sparse-only projection must store one value per entry to represent structure spread over the whole matrix, which a rank-rr factor captures with r(m+n)r(m+n) values. A low-rank-only projection cannot keep a few large, isolated entries without many extra ranks.
  • Training ablations. At a fixed budget, the hybrid beats both extremes. On CIFAR-10, adding a 5% sparse residual to low rank gains 2.25 points (74.86 vs. 72.61). On C4, the best hybrid reaches 37.94 PPL, against 38.25 for sparse-only and 114.02 for low-rank-only (Table 2a; App. E).

The two components also serve different roles across training regimes. In fine-tuning, the sparse part is a frozen anchor that retains pretrained weights at 2 bytes each, while the low-rank part carries the trainable update (14 bytes per value). This split is what makes the byte budget go far under tight constraints.

(W3: Model size for pretraining) We agree that larger-scale pretraining would strengthen the paper. A converged federated pretraining run at the 7B scale, with multiple clients, budgets, and baselines, is beyond our compute budget (the full study already required about 4,100 GH200 GPU-hours). Our evidence for scale is threefold:

  • Pretraining: two pretraining sizes (60M and 130M), where the relative gains hold and grow with model size: LLaMA-130M at 70% beats the dense LLaMA-60M.
  • Fine-tuning: fine-tuning at 1.2B parameters (Llama-3.2-1B).
  • Systems scaling: measured server-side scalability at 7B. For LLaMA-2-7B, calibration plus projection takes 2.7 s, 0.026% of one training round (App. I). The new server-cost measurements (Q1 below) include the full per-round pipeline at 7B.

We clearly state the scale limitation in the Limitations paragraph.

(W4: Theoretical guidance for the projection) The reviewer is right that the theory does not prescribe a specific mask or subspace. Its purpose is to establish convergence of factor-space local Adam with dense aggregation, and to identify which properties of the projection control the error. It does, however, guide the design in two concrete ways.

  • Projection residual. The bound depends on the projection residual δt\delta_t (and, for λagg=1\lambda_{\mathrm{agg}}=1, on ρt≤δt\rho_t\le\delta_t). Our NLP projection is exactly the minimizer of the diagonal activation-weighted distortion, characterized in Proposition B.14. Empirically, this residual is small (0.100.10 of the weight norm at the 70% reduction on LLaMA-130M) and stays flat or decreases along training.
  • Collective tangent coverage. The bound depends on κ\kappa, which measures how much of the dense gradient the clients can jointly represent. We now measure κt\kappa_t along training (see our response to Reviewer FmNX, Q1): on LLaMA-60M pretraining it stays above 1.87 throughout 110 rounds and does not decay, for both λagg=1\lambda_{\mathrm{agg}}=1 and λagg=0\lambda_{\mathrm{agg}}=0.

On changing activations and sequence lengths: the activation statistics DlD^l are re-estimated in every round at the task's context length, so the projection follows the current model and inputs. Our new calibration-domain study shows how much the projection depends on the activation source (response to Reviewer PbUJ, Q1): moving from in-domain to biomedical or code calibration data changes WikiText-2 perplexity by at most 0.57, whereas removing calibration costs 4.4–5.7 PPL. A theory that selects masks and subspaces adaptively, e.g., based on gradient alignment, is an interesting direction, which we have added to the future work (Limitations, Sec. 4; in blue).

(Q1: Server cost of projection and reconstruction vs. dense aggregation) We measured every server step per round on one GH200, using the exact projection and reconstruction code from training:

  • calibration forward pass: once per round;
  • projection: once per distinct client budget;
  • reconstruction: S+ABS+AB per returned client model;
  • averaging: identical for dense FedAvg.

We use 5 clients and the 97/3 split, and report one budget (70% reduction, homogeneous) and five distinct budgets (one client each at 20/40/50/70/90%, heterogeneous). Times are per round, in seconds, median over 5 repeats:

ModelDense FedAvgSpaRFL, λagg=1\lambda_{\mathrm{agg}}=1 (1 / 5 budgets)SpaRFL, λagg=0\lambda_{\mathrm{agg}}=0 (1 / 5 budgets)Round time
LLaMA-130M0.0110.53 / 1.510.54 / 1.56~420
Llama-3.2-1B0.0300.90 / 2.620.93 / 2.75~420–490
LLaMA-2-7B0.1403.65 / 10.023.81 / 10.8010,439

Breakdown for LLaMA-2-7B with one budget: calibration 1.36 s, projection 1.38 s, reconstruction 0.16 s per client, averaging 0.14 s. Each additional budget adds one projection (1.0–2.0 s at 7B). λagg=0\lambda_{\mathrm{agg}}=0 adds one reconstruction of the projected model sent out.

The server overhead is seconds per round, against minutes to hours of client training per round. Even at 7B with five distinct budgets it is 10.8 s, about 0.1% of a round. It grows linearly in the number of distinct budgets, not in the number of clients, since clients with the same budget share one projection. On the client side, training a projected model is up to 2.6×2.6\times faster than training the dense model (kernel benchmark in App. I). All measurements are stored with the code release. We have added this table to the paper (App. I.1, in blue).

(Q2: How much data is needed for the NLP projection, and privacy/communication) No client data or client activations are needed. The server uses a fixed set of 128 public C4 sequences, truncated to the task's context length (e.g., 256 tokens for C4, i.e., about 33K tokens). From these it computes only per-input-channel activation norms of the current dense model. The calibration set never contains client examples and is identical across clients and rounds. The projection therefore introduces no new privacy exposure and no additional communication: clients receive the projected model, exactly as in any model-heterogeneous FL method (App. M). For robustness to the choice of public data, please see the calibration-domain study in our response to Reviewer PbUJ (W2/Q1): calibrating on biomedical text or source code instead of in-domain text changes WikiText-2 perplexity by at most 0.57, so the server does not need data from the clients' domain.

(Q3: Sensitivity to the sparse/low-rank split and projection method; when is fully sparse or fully low-rank better?) The paper contains this sensitivity analysis, and it shows that the best split depends on the training regime:

  • Vision, from scratch (Table 2a). Low-rank-heavy is best. S5%/L95% gives 74.86, fully low-rank gives 72.61, and a 50/50 split gives 66.74.
  • LLM pretraining (App. E). S97%/L3% gives 37.94 PPL; fully sparse gives 38.25 (and the residual-based layer-wise rule of PbUJ W1, which selects fully sparse in every layer, gives 38.29), while fully low-rank collapses (114.02).
  • LLM fine-tuning under tight budgets (Table 2b). Keeping more sparse pretrained weights matters most. At 90%, rank r=6r=6 gives 15.67 PPL vs. 172.73 for r=128r=128. At 70%, results are robust to the rank (11.97–12.82).

So in our experiments, neither extreme is ever strictly best. Fully sparse comes close for LLM pretraining, fully low-rank comes close for vision, and the sparse+low-rank is best or tied in every regime.

On the projection method:

  • Activation space vs. weight space: activation-space projection improves C4 PPL by 14.5% over weight-space projection (App. N).
  • Mask selection: App. F ablates mask selection and initialization strategies.
  • Layer-wise allocation: new, see our response to Reviewer PbUJ, W1. A per-layer split (App. E) chosen by the projection residual selects pure sparse in every layer and does not improve over the global split (38.29 vs. 37.94), so the global split remains a robust default.

We would be glad to hear whether these clarifications and the new experiments address the reviewer's concerns.

References

[1] E. J. Candès, X. Li, Y. Ma, J. Wright. Robust Principal Component Analysis? Journal of the ACM, 2011.

[2] V. Chandrasekaran, S. Sanghavi, P. A. Parrilo, A. S. Willsky. Rank-Sparsity Incoherence for Matrix Decomposition. SIAM Journal on Optimization, 2011.

[3] Y. Li et al. LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation. ICML, 2023.

[4] A. Han et al. SLTrain: a Sparse plus Low-Rank Approach for Parameter and Memory Efficient Pretraining. NeurIPS, 2024.

Reviewer_PbUJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles a super practical problem in FL called the "no-client-fits" regime, where not a single participating device has enough memory to hold and optimize the full global Transformer model. To get around this bottleneck, the work introduces SpaRFL, a neat framework where the server keeps a dense global model but sends out custom-sized sparse plus low-rank projections to each client based on its specific byte-level memory budget. Clients just train these lighter, projected versions locally and send the updates back to the server, which then reconstructs them into the dense space for standard aggregation. Across a bunch of vision and NLP benchmarksm, covering both pretraining from scratch and fine-tuning, SpaRFL outperforms existing heterogeneous FL baselines, and even lets a compressed LLaMA-130M model beat a dense LLaMA-60M baseline under the same memory budget.

Critical strengths
  • The writing of the paper, not just introduction, is very clear and easy to follow.

  • Framing the "no-client-fits" regime around the actual persistent training state in bytes (counting weights, gradients, and AdamW optimizer states) instead of just counting trainable parameters is a very interesting idea. Under AdamW, a trainable parameter takes 14 bytes while a frozen half-precision weight only takes 2 bytes, so this byte-level accounting reflects how hardware memory actually runs out in the real world.

  • The way SpaRFL adapts its projection strategy to the task and training stage is quite thoughtful. Using weight-space SVD plus top-k residual pruning works great for vision, while switching to an activation-aware projection with a reversed decomposition order for NLP pretraining (low-rank first, sparse second) versus fine-tuning (sparse first, low-rank second) makes complete sense given how outlier activations behave in LLMs.

+The experimental evaluation is very comprehensive and solid. Testing everything from ViT-Tiny on CIFAR-10 to RoBERTa on GLUE and LLaMA pretraining on C4 shows the method is genuinely versatile. Seeing SpaRFL stay within 1% of the dense baseline on MRPC at a massive 90% budget reduction, and seeing LLaMA-130M at a 70% budget reduction beat a dense LLaMA-60M model, are fantastic headline results.

  • Because the server reconstructs the sparse and low-rank updates back into a dense matrix, SpaRFL acts as a clean interface that plays nicely with off-the-shelf server optimizers like FedAvg, FedAdam, and FedAdagrad, and even stacks effortlessly with 4-bit NormalFloat quantization (Q-SpaRFL) to squeeze out extra performance under tight budgets.
Critical weaknesses
  • The byte budget split γ\gamma between the sparse and low-rank components is fixed globally across all Transformer layers and stays static throughout the entire training run. Yet, Figure 5 in the appendix clearly shows that different layer types (like attention Q/K/V vs. MLP down-projections) have totally different spectral and outlier behaviors. Keeping γ\gamma uniform across the whole network feels like a missed opportunity for a layer-wise or dynamic allocation.

  • The activation-aware NLP projection assumes the server can just use 128 sequences from C4 as public calibration data. While that works fine for standard benchmarks like GLUE or WikiText-2, cross-institutional FL often deals with specialized domains (like clinical notes in hospitals, as mentioned in the introduction) where generic web text might not capture the right activation outliers at all.

Questions for authors
  • In cross-silo scenarios like healthcare or finance, client data distributions can be miles apart from public C4 text. How sensitive is the activation-aware projection if the server's public calibration dataset DcalD_{\text{cal}} has a massive domain shift compared to the private client datasets?

  • In the factor-space Adam setup, the formulation notes that local optimizer moments can be reset every round because the sparse support Ωt,i\Omega_{t,i} and low-rank factors change after each server projection. In the actual implementation, are the client AdamW moments completely wiped at the start of every communication round, and does that cause any warm-up instability when local steps EE are small?

4 comments

We thank the reviewer for the thoughtful and very positive review. We are glad the reviewer found the byte-level accounting a realistic reflection of how training memory runs out, appreciated the regime-dependent design of the projection, and found the evaluation comprehensive. We address the weaknesses and questions below.

(W1: Uniform sparse/low-rank split across layers) Thank you for this suggestion. We used a single global split γ\gamma to keep the analysis controlled and the method free of per-layer tuning. We agree that the spectral and outlier behavior differs across layer types (Figure 5), so we implemented and tested a layer-wise allocation.

For each layer, given the activation-weighted weight WlDlW^lD^l, we choose the sparse fraction from a grid {0.80,0.90,0.95,0.97,0.99,1.0}\{0.80,0.90,0.95,0.97,0.99,1.0\} that minimizes the activation-weighted projection residual at that layer's byte budget. One exact SVD per layer makes this cheap: for every candidate rank, the residual after the best top-kk sparse completion follows directly. The total byte budget per layer is unchanged, so the comparison is at matched memory. We re-select the split in every round, which also makes the allocation dynamic over training.

C4 LLaMA-60M, 70%C4 PPL$\downarrow$
Global split (S 97% / L 3%)37.94
Pure sparse (S 100%), all layers38.25
Layer-wise split (re-selected every round)38.29

The layer-wise rule selected the pure sparse allocation (sparse fraction 1.0) in every layer type (q, k, v, o, gate, up, down) and every block, at all rounds we inspected, from the first to the last. Accordingly, it matches the pure-sparse ablation of the paper (38.29 vs. 38.25) and is 0.35 PPL worse than the global split. So, in pretraining, a per-layer allocation chosen by the projection residual collapses to pure sparse and loses the benefit of the low-rank component. The criterion is myopic: it measures how well the current weights are represented, and ignores the trainable directions that the low-rank factors add for the following local updates. The global split is therefore a robust default, and learning the allocation from the training signal itself is a promising direction. We have added this experiment to the paper (App. E) and mentioned learned allocation as future work (Limitations, Sec. 4; in blue).

(W2 / Q1: Sensitivity to the calibration domain) This is an important practical point, and we thank the reviewer for raising it. Two remarks on the setting first:

  • The calibration set is small, public, and generic. The activation-aware projection uses only 128 public sequences, and only their per-channel activation norms DlD^l .
  • Public in-domain corpora exist for the domains in question: e.g., PubMed or PMC open-access text for clinical and biomedical settings, or public filings for finance. A server can draw calibration data from them without touching client data.

To quantify sensitivity, we fine-tuned Llama-3.2-1B on WikiText-2 at the 90% budget reduction, where the frozen sparse support is selected entirely by the activation-weighted score. We varied only the calibration corpus (128 sequences each):

Calibration dataWikiText-2 PPL$\downarrow$
C4 (paper default, generic web text)15.67
WikiText-2 validation (in-domain, held out)16.33
PubMed articles (biomedical)16.81
Python code16.90
No calibration (weight-space projection)21.34

The projection is robust to the calibration domain. In-domain, biomedical, and code calibration lie within 0.57 PPL of each other: moving from in-domain text to a strong domain shift, biomedical text or source code, costs only 0.48–0.57 PPL. Dropping calibration altogether costs 4.4–5.7 PPL. What matters is therefore having activation statistics at all, much more than their domain, and generic web text (C4, our default) works well. In this study, the clients' task is fixed (WikiText-2) and only the calibration data varies. The complementary case, in which the clients train on biomedical or code data and the server calibrates on generic or domain-matched public text, remains to be evaluated and is a natural next step. We have added this study to the paper (App. M, in blue).

(Q2: Resetting AdamW moments every round) Yes. In our implementation, every client creates a fresh AdamW optimizer at the start of each round, so the first and second moments are zero-initialized and Adam's bias correction applies from the first local step. This is common practice in FL with local adaptive optimizers. Clients are typically stateless across rounds, and their optimizer state is not carried over or aggregated [1, 2, 3]. In SpaRFL, carrying the moments over would not even be well defined: after each server projection, the sparse support and the factor coordinates (A,B)(A,B) change, so last round's moments refer to different coordinates.

On warm-up: the learning rate is warmed up linearly over the first 5% of rounds (5 of 110 rounds in C4 pretraining), held constant within a round, and then decayed with a cosine schedule. Within each round, Adam's bias correction keeps the first local steps well-scaled. In C4 pretraining, each round consists of 75 local AdamW steps (1200 micro-batches, 16 accumulation steps). We did not observe instabilities at round boundaries: at the 70% reduction, the server-side C4 validation loss decreases in 109 of the 110 rounds, and the single increase is 0.0002 nats. We have added these details to the paper (App. A, in blue).

We thank the reviewer again for the encouraging assessment, and would be happy to clarify any remaining points.

References

[1] P. Kairouz et al. Advances and Open Problems in Federated Learning. Foundations and Trends in Machine Learning, 14(1–2), 2021.

[2] S. J. Reddi et al. Adaptive Federated Optimization. ICLR, 2021.

[3] J. Wang et al. A Field Guide to Federated Optimization. arXiv:2107.06917, 2021.

Thanks for the detailed experiment results and clarification. I have no further concerns about this paper and I keep my positive score.

Thank you for your constructive feedback and positive assessment of our work.

Reviewer_FmNXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies FL of transformer models in heterogenous setting where clients are unable to fit a whole model. In every round, server compresses the dense model in client-specific sparse and low-rank model that fits client's memory. Clients train these models, and server then reconstructs the dense versions and aggregate them. They provide theoretical convergence and empirical results on various transformer models under different settings.

Critical strengths
  1. Defining client budget in bytes is practical and separates trainable weights from frozen effectively.
  2. The paper is clear and easy to follow.
Critical weaknesses

W1. The main claim of the paper is that the server keeps a dense model. However, all the experiments update the server by averaging the clients' compressed models with λ_agg = 1. This effectively means that the server model itself becomes compressed, since whatever the compression drops is lost every round. Comparatively, the paper's own analysis notes that delta aggregation (λ_agg = 0) avoids this error but it is never run, and this limitation is not discussed.

W2. The baseline used for comparison largely fail to train in Table 1. I suggest chosing a setting mentioned in those baseline where they converge for comparison otherwise, it undermines the credibility of these results.

W3. The main PEFT results with same budgets are in Table 9 which shows that SpaRFL is slightly worse on all tasks compared to RoSA.

W4. A very similar work is missing from comparison, namely FedLMT [1]. Can you compare your technique against this?

[1] Liu, Jiahao, et al. "FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees." ICML. 2024.

Questions for authors

Q1. The convergence only guarantees reaching a neighbours solution by assumption B.6. Have this condition been validated empirically?

Q2. How does a FL baseline with frozen base model and LoRA, at the same memory budget compare with the proposed technique?

4 comments

We thank the reviewer for the careful review, and for highlighting that defining the client budget in bytes is practical and cleanly separates trainable from frozen state, and that the paper is clear and easy to follow. We address each point below; all new results have been added to the revised paper (in blue).

(W1: Dense server model and λagg=1\lambda_{\mathrm{agg}}=1) We agree that the paper should have reported delta aggregation and discussed the trade-off. In the paper, we used λagg=1\lambda_{\mathrm{agg}}=1, i.e., the server averages the models the clients return: this is the standard FedAvg aggregation used by most FL work, and we applied it to all methods. Thanks to the reviewer's comment, we now extend the study to delta aggregation (λagg=0\lambda_{\mathrm{agg}}=0), for SpaRFL and for all baselines (results below). First, we clarify what the dense server provides.

  • Even under λagg=1\lambda_{\mathrm{agg}}=1, the aggregate leaves the clients' compressed class. With λagg=1\lambda_{\mathrm{agg}}=1, Wt+1=∑iqi(St,i,E+At,i,EBt,i,E)\mathcal{W}_{t+1}=\sum_i q_i(S_{t,i,E}+A_{t,i,E}B_{t,i,E}). Each client trains on its own data, so even clients with the same budget, which receive the same factors (A,B)(A,B), return different ones. Writing At,i,E=A+ΔAiA_{t,i,E}=A+\Delta A_i and Bt,i,E=B+ΔBiB_{t,i,E}=B+\Delta B_i, the averaged low-rank part is ∑iqiAt,i,EBt,i,E=AB+AΔB‾+ΔA‾B+∑iqiΔAiΔBi\sum_i q_iA_{t,i,E}B_{t,i,E}=AB+A\overline{\Delta B}+\overline{\Delta A}B+\sum_i q_i\Delta A_i\Delta B_i. The first three terms have rank up to 2r2r and the last adds up to nrnr, so the aggregate generally exceeds the rank of every client. Storing it in any client format would require a further lossy projection; the dense server stores it exactly.
  • Delta aggregation, which the reviewer asks for, requires the dense server in all cases. With λagg=0\lambda_{\mathrm{agg}}=0, Wt+1=Wt+∑iqi[(St,i,E+At,i,EBt,i,E)−(St,i,0+At,i,0Bt,i,0)]\mathcal{W}_{t+1}=\mathcal{W}_t+\sum_i q_i\big[(S_{t,i,E}+A_{t,i,E}B_{t,i,E})-(S_{t,i,0}+A_{t,i,0}B_{t,i,0})\big]. The server adds only the clients' updates and keeps what the projections drop, which by definition no client can hold, in pretraining and fine-tuning alike.
  • The dense server costs the clients nothing. The memory constraint applies to clients, not the server. In fine-tuning, the full pretrained weights exist only on the server; clients only receive and store their projected models.
  • What is discarded per round is small and does not grow. The reviewer is right that λagg=1\lambda_{\mathrm{agg}}=1 drops the projection residual of each round. Along the LLaMA-130M trajectory at the 70% reduction (App. B.6), this activation-weighted residual is 0.100.10 of the weight norm and decreases mildly (0.1130.113 at round 20, 0.0980.098 at round 80).
  • The two interfaces trade off different errors. Our analysis (Thm. B.10, Cor. B.11–B.12) covers both: λagg=0\lambda_{\mathrm{agg}}=0 removes the mean projection displacement ρt\rho_t, while λagg=1\lambda_{\mathrm{agg}}=1 keeps the server consistent with the parameterization the clients optimize. SpaRFL supports both without any change to the clients.

New results. We ran λagg=0\lambda_{\mathrm{agg}}=0 with otherwise identical settings. The λagg=1\lambda_{\mathrm{agg}}=1 column is from the paper, except SST-2 (20 rounds) and MRPC/QNLI (client learning rate 5⋅10−55\cdot10^{-5} in both columns):

Settingλagg=1\lambda_{\mathrm{agg}}=1λagg=0\lambda_{\mathrm{agg}}=0
WikiText-2, homogeneous 70% (PPL$\downarrow$)11.9711.76
WikiText-2, homogeneous 90% (PPL$\downarrow$)15.6714.24
WikiText-2, heterogeneous (PPL$\downarrow$)13.09 ± 0.2711.82 ± 0.02
SST-2, heterogeneous, 20 rounds (Acc.↑\uparrow)93.16 ± 0.2493.73 ± 0.58
MRPC, heterogeneous (F1/Acc.↑\uparrow)88.01 ± 5.92 / 81.54 ± 11.4291.67 ± 0.49 / 88.40 ± 0.75
QNLI, heterogeneous (Acc.↑\uparrow)80.59 ± 19.5492.43 ± 0.08
C4 LLaMA-60M, 50% (PPL$\downarrow$)35.7036.13
C4 LLaMA-60M, 70% (PPL$\downarrow$)37.9444.85

λagg=0\lambda_{\mathrm{agg}}=0 helps most when fine-tuning under tight budgets (90% reduction, heterogeneous). This is the case the reviewer describes: with λagg=1\lambda_{\mathrm{agg}}=1, the pretrained weights that the frozen masks drop are replaced by their low-rank approximation, whereas with λagg=0\lambda_{\mathrm{agg}}=0 the server keeps them. In pretraining from scratch, the order reverses: no pretrained information is lost, and with λagg=0\lambda_{\mathrm{agg}}=0 the server keeps components that no client can update. We therefore recommend λagg=0\lambda_{\mathrm{agg}}=0 for fine-tuning and λagg=1\lambda_{\mathrm{agg}}=1 for pretraining. We have added these results and the trade-off to the paper (App. O, and the Aggregation Protocol paragraph in Sec. 4), and we now state λagg\lambda_{\mathrm{agg}} for every experiment.

(W2: Baselines fail to train in Table 1) We thank the reviewer for pushing on this point.

First, the context: the setting in Table 1 is the no-client-fits regime by design. In the original HeteroFL, FjORD, and FIARSE evaluations, the largest client tier trains the full model. This regime is the subject of the paper.

In the paper, all methods use the same server rule: standard FedAvg over the clients' reconstructed dense models (λagg=1\lambda_{\mathrm{agg}}=1). This is where the baselines fail. A coordinate outside a client's sub-model enters the average as zero, and in the no-client-fits regime many coordinates are held by few or no clients, so they are pulled toward zero in every round. SpaRFL does not suffer from this, since its low-rank component covers every coordinate. With delta aggregation (λagg=0\lambda_{\mathrm{agg}}=0), the server keeps each coordinate's value and adds only the updates of the clients that train it.

We have re-run all the baselines with λagg=0\lambda_{\mathrm{agg}}=0 on the Table 1 settings:

MethodSST-2, 20 rounds (Acc.↑\uparrow)MRPC (F1/Acc.↑\uparrow)QNLI (Acc.↑\uparrow)WikiText-2 (PPL ↓\downarrow)
HeteroFL, λagg=1\lambda_{\mathrm{agg}}=178.33 ± 1.4381.22 / 68.3855.551599.17
HeteroFL, λagg=0\lambda_{\mathrm{agg}}=092.05 ± 0.7082.16 ± 0.57 / 72.47 ± 2.2887.78 ± 2.6915.59
FjORD, λagg=1\lambda_{\mathrm{agg}}=178.17 ± 0.8381.69 / 69.5358.071464.94
FjORD, λagg=0\lambda_{\mathrm{agg}}=091.25 ± 0.7580.47 ± 2.79 / 70.18 ± 0.7989.16 ± 0.3615.56
FIARSE, λagg=1\lambda_{\mathrm{agg}}=182.57 ± 0.3081.71 / 69.6165.92464.78
FIARSE, λagg=0\lambda_{\mathrm{agg}}=091.40 ± 0.2384.25 ± 2.08 / 75.25 ± 4.4390.12 ± 0.2514.97
SpaRFL, λagg=1\lambda_{\mathrm{agg}}=193.16 ± 0.2491.57 / 88.1589.0613.09
SpaRFL, λagg=0\lambda_{\mathrm{agg}}=093.73 ± 0.5891.67 ± 0.49 / 88.40 ± 0.7592.43 ± 0.0811.82

(On MRPC/QNLI, all λagg=0\lambda_{\mathrm{agg}}=0 runs use client learning rate 5⋅10−55\cdot10^{-5}; at the paper's 3⋅10−43\cdot10^{-4}, every λagg=0\lambda_{\mathrm{agg}}=0 run, including SpaRFL, stays at the majority class.)

SpaRFL remains the best method on all four tasks, under either aggregation rule. With λagg=0\lambda_{\mathrm{agg}}=0 the baselines improve substantially, but SpaRFL still leads the best baseline on every task, e.g., by 7.4 F1 on MRPC and 3.2 PPL on WikiText-2. It also stays best when the baselines use the per-coordinate rule of the original HeteroFL/FIARSE papers.

In pretraining from scratch (C4, LLaMA-60M, 70%, the Figure 3 setting), the baselines do not converge with λagg=0\lambda_{\mathrm{agg}}=0 (perplexity above 4,000, vs. 37.94 for SpaRFL): coordinates outside every client's sub-model are never updated and keep their random initialization. SpaRFL avoids this, since its low-rank component updates every coordinate.

We have added these results to the paper (App. P, referenced from the Table 1 discussion, in blue). We are grateful to the reviewer: this makes the comparison considerably more credible.

(W3: PEFT comparison, Table 9) We agree with the reading of Table 9 at the 70% budget, and we report it as such in the paper: when a pretrained base fits in client memory, RoSA is a strong method. We would like to add three points of context.

  • The gap at 70% is small: 0.0590.059 PPL on WikiText-2, and 0.20.2 F1 on MRPC.
  • At 90%, the order reverses: Q-SpaRFL reaches 12.64 PPL on WikiText-2, against 13.20 for QRoSA and 13.57 for QLoRA. Unquantized RoSA is not even applicable there: a frozen half-precision base alone costs 2P2P bytes, which bounds any unquantized PEFT method to at most an 85.71% reduction under our accounting.
  • PEFT cannot pretrain: SpaRFL is not a PEFT method. It also pretrains from scratch, where frozen-base methods fail (C4 PPL above 1500 for QLoRA and RoSA-style diagnostics, vs. 37.94 for SpaRFL).

In short, SpaRFL is competitive with the strongest PEFT method where PEFT applies, and it remains applicable in the stricter regimes where PEFT does not.

(W4: Comparison with FedLMT) Thank you for pointing us to FedLMT; we now discuss and compare against it. FedLMT trains a single pre-factorized low-rank model UV⊤UV^\top with the same architecture on all clients, sized for the weakest one, and aggregates the factors directly. SpaRFL differs in three ways: it keeps a dense server model, re-projects it per client and per round, and adds a sparse component to the low-rank one.

To reach our budgets, we factorize all layers SpaRFL projects, since FedLMT's hybrid design keeps some layers dense, which is infeasible at a 50–70% reduction. In the heterogeneous settings, FedLMT trains one model sized for the tightest client tier (90%), as in the paper. On C4 at the 70% reduction we tuned the Frobenius-decay coefficient over 10−5,10−4,10−3\\{10^{-5},10^{-4},10^{-3}\\} (41.06 / 44.33 / 81.81 PPL) and report the best.

SettingFedLMTSpaRFL
C4 LLaMA-60M, 70% (PPL$\downarrow$)41.0637.94
C4 LLaMA-60M, 50% (PPL$\downarrow$)41.8335.70
WikiText-2, homogeneous 70% (PPL$\downarrow$)1061.2211.97
WikiText-2, heterogeneous (PPL$\downarrow$)1610.4413.09 (λagg=1\lambda_{\mathrm{agg}}=1) / 11.82 (λagg=0\lambda_{\mathrm{agg}}=0)
MRPC, heterogeneous (F1/Acc.↑\uparrow)81.22 / 68.3891.57 / 88.15

SpaRFL outperforms FedLMT in every setting. In fine-tuning, a low-rank-only model cannot retain the pretrained weights (WikiText-2 stays around 10310^3 PPL); SpaRFL's sparse component keeps them. We have added FedLMT to the paper (Sec. 2, Sec. 4, Figure 3, App. Q; in blue).

(Q1: Validation of Assumption B.6) Good question. Assumption B.6 asks that the client tangent spaces, collectively, capture the squared dense gradient norm up to a constant κ>0\kappa>0. We measured this directly along the LLaMA-60M C4 trajectory (λagg∈0,1\lambda_{\mathrm{agg}}\in\\{0,1\\}, 70% budget). At saved checkpoints, we compute the dense gradient Gt=∇Φ(Wt)G_t=\nabla\Phi(\mathcal{W}_t) on held-out C4 batches, build every client's projected model exactly as in training, and evaluate κt=∑iqi∣Jt,i∗Gt∣2/∣Gt∣2\kappa_t=\sum_i q_i\\|J_{t,i}^{\ast}G_t\\|^2/\\|G_t\\|^2, with J∗G=(PΩG,GB⊤,A⊤G)J^{\ast}G=(P_\Omega G, GB^\top, A^\top G). As an isotropic reference, we report the same ratio for a random direction with the gradient's per-layer norms.

Roundκt\kappa_t, λagg=1\lambda_{\mathrm{agg}}=1κtproj\kappa_t^{\mathrm{proj}}, λagg=1\lambda_{\mathrm{agg}}=1isotropic ref., λagg=1\lambda_{\mathrm{agg}}=1κt\kappa_t, λagg=0\lambda_{\mathrm{agg}}=0κtproj\kappa_t^{\mathrm{proj}}, λagg=0\lambda_{\mathrm{agg}}=0isotropic ref., λagg=0\lambda_{\mathrm{agg}}=0
101.872.081.702.422.651.52
303.143.642.513.033.442.97
502.332.672.914.645.354.02
702.863.343.084.344.944.15
902.352.662.984.014.454.06
1102.442.772.923.744.103.86

(κt\kappa_t: whole model; κtproj\kappa_t^{\mathrm{proj}}: projected layers only; isotropic ref.: the same ratio for a generic direction with the same per-layer norms, projected layers.)

Over all 110 rounds and both interfaces, κt\kappa_t never falls below 1.87, and it does not decay with training. This is the behavior Assumption B.6 requires: a constant κ>0\kappa>0 along the trajectory. The coverage is also uniform across directions: on the projected layers, the true gradient is covered as well as a generic direction of the same per-layer norm (0.8×0.8\times to 1.7×1.7\times the isotropic reference over all saved rounds). The guarantee therefore does not rely on the gradient falling into a favorable subspace; it holds because the clients' sparse supports and factors jointly span enough of each layer, whatever direction the gradient takes. The values exceed 1 because J∗J^{\ast} also scales the gradient by the factor norms (GB⊤GB^\top, A⊤GA^\top G). We added this to the paper (App. B.7, in blue).

We evaluate the coverage at the projection point (k=0k=0), i.e., for the tangent spaces the server hands out. The sufficient certificate of Proposition B.7 is zero by construction for a single client tier (rank r<min⁡(m,n)r<\min(m,n) and a sparse mask), which is why the trajectory quantity κt\kappa_t is the informative check.

(Q2: LoRA with a frozen base at the same budget) This comparison is in App. G (Table 9 of the submission), at matched persistent-state budgets:

  • LoRA-based baselines: FLASC (federated LoRA) and Fed-PLoRA (heterogeneous federated LoRA).
  • Frozen-base PEFT baselines: QLoRA and RoSA.

At the 70% reduction, SpaRFL outperforms FLASC on all four tasks, and Fed-PLoRA on MRPC, QNLI, and WikiText-2. Fed-PLoRA and QLoRA are better on SST-2. At the 90% reduction, unquantized LoRA is infeasible (see W3), and Q-SpaRFL improves on QLoRA (12.64 vs. 13.57 PPL). For pretraining, a frozen random base leaves LoRA-style methods above 1500 PPL on C4.

We would be glad to hear whether these address the reviewer's concerns.

We thank the reviewer again for their comments. As the discussion period is approaching its end (Nov 18 AoE), we would be glad to hear back from the reviewer, to understand if our responses addressed their concerns. If any concerns remain unresolved, we would be happy to respond before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.