- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors study federated Transformer training in the no-client-fits regime, where no participating client can store and train the full dense global model within its local memory budget. To address this setting, they propose SpaRFL (Sparse Low-Rank Federated Learning), a client-specific one-shot sparse+low-rank projection method that constructs trainable client models under heterogeneous memory constraints while preserving a dense server model.
Critical strengths
- The paper is well-written and easy to follow.
- The motivation for conducting this work is clear, i.e., to address the no-client-fits regime problem, where no participating client can store and train the full dense global model within its local memory budget.
Critical weaknesses
- The novelty of this work is limited. The difference between this work and FedLRS seems to be that the clients have heterogeneous memory budgets, but the method is similar, i.e., training sparse and low-rank components.
- The motivation of the proposed method is not clear. In lines 195- 196, "represent each client layer as the sum of a low-rank component and a sparse residual.", why are they split into these two? Could the authors provide theoretical or experimental explanations? Or is it just a follow-up to FedLRS?
- The experimental model used for pretraining is too small in size (i.e., only 120M). The authors should conduct experiments on a larger model size, e.g., 7B or larger.
- The theory gives little help in choosing the projection. The analysis supports local training in a projected model, but it does not explain when a particular sparse mask or low-rank space keeps the useful Transformer updates, especially when activations or sequence lengths change.
Questions for authors
- What are the server costs for projection and rebuilding compared with normal dense aggregation, especially for large language-model layers?
- How much client data or activation information is needed for the NLP projection, and can this be computed without new privacy or communication problems?
- How sensitive are the results to the sparse/low-rank budget split and to the projection method? When is a fully sparse or fully low-rank method better?
6 comments
Part II
(W4: Theoretical guidance for the projection) The reviewer is right that the theory does not prescribe a specific mask or subspace. Its purpose is to establish convergence of factor-space local Adam with dense aggregation, and to identify which properties of the projection control the error. It does, however, guide the design in two concrete ways.
- Projection residual. The bound depends on the projection residual (and, for , on ). Our NLP projection is exactly the minimizer of the diagonal activation-weighted distortion, characterized in Proposition B.14. Empirically, this residual is small ( of the weight norm at the 70% reduction on LLaMA-130M) and stays flat or decreases along training.
- Collective tangent coverage. The bound depends on , which measures how much of the dense gradient the clients can jointly represent. We now measure along training (see our response to Reviewer FmNX, Q1): on LLaMA-60M pretraining it stays above 1.87 throughout 110 rounds and does not decay, for both and .
On changing activations and sequence lengths: the activation statistics are re-estimated in every round at the task's context length, so the projection follows the current model and inputs. Our new calibration-domain study shows how much the projection depends on the activation source (response to Reviewer PbUJ, Q1): moving from in-domain to biomedical or code calibration data changes WikiText-2 perplexity by at most 0.57, whereas removing calibration costs 4.4–5.7 PPL. A theory that selects masks and subspaces adaptively, e.g., based on gradient alignment, is an interesting direction, which we have added to the future work (Limitations, Sec. 4; in blue).
(Q1: Server cost of projection and reconstruction vs. dense aggregation) We measured every server step per round on one GH200, using the exact projection and reconstruction code from training:
- calibration forward pass: once per round;
- projection: once per distinct client budget;
- reconstruction: per returned client model;
- averaging: identical for dense FedAvg.
We use 5 clients and the 97/3 split, and report one budget (70% reduction, homogeneous) and five distinct budgets (one client each at 20/40/50/70/90%, heterogeneous). Times are per round, in seconds, median over 5 repeats:
| Model | Dense FedAvg | SpaRFL, (1 / 5 budgets) | SpaRFL, (1 / 5 budgets) | Round time |
|---|---|---|---|---|
| LLaMA-130M | 0.011 | 0.53 / 1.51 | 0.54 / 1.56 | ~420 |
| Llama-3.2-1B | 0.030 | 0.90 / 2.62 | 0.93 / 2.75 | ~420–490 |
| LLaMA-2-7B | 0.140 | 3.65 / 10.02 | 3.81 / 10.80 | 10,439 |
Breakdown for LLaMA-2-7B with one budget: calibration 1.36 s, projection 1.38 s, reconstruction 0.16 s per client, averaging 0.14 s. Each additional budget adds one projection (1.0–2.0 s at 7B). adds one reconstruction of the projected model sent out.
The server overhead is seconds per round, against minutes to hours of client training per round. Even at 7B with five distinct budgets it is 10.8 s, about 0.1% of a round. It grows linearly in the number of distinct budgets, not in the number of clients, since clients with the same budget share one projection. On the client side, training a projected model is up to faster than training the dense model (kernel benchmark in App. I). All measurements are stored with the code release. We have added this table to the paper (App. I.1, in blue).
(Q2: How much data is needed for the NLP projection, and privacy/communication) No client data or client activations are needed. The server uses a fixed set of 128 public C4 sequences, truncated to the task's context length (e.g., 256 tokens for C4, i.e., about 33K tokens). From these it computes only per-input-channel activation norms of the current dense model. The calibration set never contains client examples and is identical across clients and rounds. The projection therefore introduces no new privacy exposure and no additional communication: clients receive the projected model, exactly as in any model-heterogeneous FL method (App. M). For robustness to the choice of public data, please see the calibration-domain study in our response to Reviewer PbUJ (W2/Q1): calibrating on biomedical text or source code instead of in-domain text changes WikiText-2 perplexity by at most 0.57, so the server does not need data from the clients' domain.
Part III
(Q3: Sensitivity to the sparse/low-rank split and projection method; when is fully sparse or fully low-rank better?) The paper contains this sensitivity analysis, and it shows that the best split depends on the training regime:
- Vision, from scratch (Table 2a). Low-rank-heavy is best. S5%/L95% gives 74.86, fully low-rank gives 72.61, and a 50/50 split gives 66.74.
- LLM pretraining (App. E). S97%/L3% gives 37.94 PPL; fully sparse gives 38.25 (and the residual-based layer-wise rule of PbUJ W1, which selects fully sparse in every layer, gives 38.29), while fully low-rank collapses (114.02).
- LLM fine-tuning under tight budgets (Table 2b). Keeping more sparse pretrained weights matters most. At 90%, rank gives 15.67 PPL vs. 172.73 for . At 70%, results are robust to the rank (11.97–12.82).
So in our experiments, neither extreme is ever strictly best. Fully sparse comes close for LLM pretraining, fully low-rank comes close for vision, and the sparse+low-rank is best or tied in every regime.
On the projection method:
- Activation space vs. weight space: activation-space projection improves C4 PPL by 14.5% over weight-space projection (App. N).
- Mask selection: App. F ablates mask selection and initialization strategies.
- Layer-wise allocation: new, see our response to Reviewer PbUJ, W1. A per-layer split (App. E) chosen by the projection residual selects pure sparse in every layer and does not improve over the global split (38.29 vs. 37.94), so the global split remains a robust default.
We would be glad to hear whether these clarifications and the new experiments address the reviewer's concerns.
References
[1] E. J. Candès, X. Li, Y. Ma, J. Wright. Robust Principal Component Analysis? Journal of the ACM, 2011.
[2] V. Chandrasekaran, S. Sanghavi, P. A. Parrilo, A. S. Willsky. Rank-Sparsity Incoherence for Matrix Decomposition. SIAM Journal on Optimization, 2011.
[3] Y. Li et al. LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation. ICML, 2023.
[4] A. Han et al. SLTrain: a Sparse plus Low-Rank Approach for Parameter and Memory Efficient Pretraining. NeurIPS, 2024.
Part I
We thank the reviewer for the review, and for noting that the paper is well written and that the motivation, training in the no-client-fits regime where no client can store and train the dense global model, is clear. We respond to each point below.
(W1: Novelty relative to FedLRS) We respectfully disagree that the difference to FedLRS reduces to heterogeneous budgets. Sparse+low-rank parameterizations are an established tool (RoSA, LoSparse, SLTrain, FedLRS), and we do not claim the parameterization itself as our contribution. Our contributions are:
Empirically, these differences are large. At matched budgets on C4, SpaRFL improves over FedLRS by 20.1 PPL on LLaMA-60M (37.94 vs. 58.03) and by 9.4 PPL on LLaMA-130M (31.56 vs. 40.91).
(W2: Why sparse + low-rank?) The rationale has three parts.
The two components also serve different roles across training regimes. In fine-tuning, the sparse part is a frozen anchor that retains pretrained weights at 2 bytes each, while the low-rank part carries the trainable update (14 bytes per value). This split is what makes the byte budget go far under tight constraints.
(W3: Model size for pretraining) We agree that larger-scale pretraining would strengthen the paper. A converged federated pretraining run at the 7B scale, with multiple clients, budgets, and baselines, is beyond our compute budget (the full study already required about 4,100 GH200 GPU-hours). Our evidence for scale is threefold:
We clearly state the scale limitation in the Limitations paragraph.