- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MAPO, which flattens the entire model gradient and reshapes it into a matrix (model-independence), and performs parametric updates using a randomly regenerated reconstruction vector and a trainable projection vector in each iteration. The client performs optimization within a randomly sampled subspace and uploads only the random seed, thereby achieving communication efficiency.
Strengths
The method is very concise and easy to understand, and it avoids the need to select low rank on a layer-by-layer basis. Furthermore, the paper features robust and comprehensive experiments, and the results demonstrate the effectiveness of MAPO.
Weaknesses
- The paper points out the drawbacks of layer-by-layer rank selection and emphasizes model-independence. In fact, for both low-rank matrix decomposition and low-rank tensor decomposition, each layer of most models can be viewed as a matrix or tensor. Therefore, such methods are also model-agnostic. It would be helpful if the authors could include a comparison with low-rank matrix decomposition and low-rank tensor decomposition methods (which are layer-by-layer decomposition approaches). Additionally, the authors should compare MAPO with a layer-by-layer decomposition version of MAPO to better demonstrate its effectiveness.
- Are the other baselines in the experimental section the result of thorough hyperparameter tuning? How were the hyperparameters for the comparison baselines selected? Furthermore, how do accuracy and communication overhead behave when clients join or leave the system? If the number of local training iterations per client is altered, would MAPO’s advantages still hold?
- The concept of MAPO was not first proposed in this paper. Therefore, I suggest that the authors clearly define the scope of their contributions.
Critical strengths
The method is very concise and easy to understand, and it avoids the need to select low rank on a layer-by-layer basis. Furthermore, the paper features robust and comprehensive experiments, and the results demonstrate the effectiveness of MAPO.
Critical weaknesses
- The paper points out the drawbacks of layer-by-layer rank selection and emphasizes model-independence. In fact, for both low-rank matrix decomposition and low-rank tensor decomposition, each layer of most models can be viewed as a matrix or tensor. Therefore, such methods are also model-agnostic. It would be helpful if the authors could include a comparison with low-rank matrix decomposition and low-rank tensor decomposition methods (which are layer-by-layer decomposition approaches). Additionally, the authors should compare MAPO with a layer-by-layer decomposition version of MAPO to better demonstrate its effectiveness.
- Are the other baselines in the experimental section the result of thorough hyperparameter tuning? How were the hyperparameters for the comparison baselines selected? Furthermore, how do accuracy and communication overhead behave when clients join or leave the system? If the number of local training iterations per client is altered, would MAPO’s advantages still hold?
- The concept of MAPO was not first proposed in this paper. Therefore, I suggest that the authors clearly define the scope of their contributions.
Questions for authors
- The paper points out the drawbacks of layer-by-layer rank selection and emphasizes model-independence. In fact, for both low-rank matrix decomposition and low-rank tensor decomposition, each layer of most models can be viewed as a matrix or tensor. Therefore, such methods are also model-agnostic. It would be helpful if the authors could include a comparison with low-rank matrix decomposition and low-rank tensor decomposition methods (which are layer-by-layer decomposition approaches). Additionally, the authors should compare MAPO with a layer-by-layer decomposition version of MAPO to better demonstrate its effectiveness.
- Are the other baselines in the experimental section the result of thorough hyperparameter tuning? How were the hyperparameters for the comparison baselines selected? Furthermore, how do accuracy and communication overhead behave when clients join or leave the system? If the number of local training iterations per client is altered, would MAPO’s advantages still hold?
- The concept of MAPO was not first proposed in this paper. Therefore, I suggest that the authors clearly define the scope of their contributions.
5 comments
Response to Reviewer vYix (2/2)
W2: Tuning of the baselines, clients joining or leaving, and local iterations.
Response to W2:
(a) Tuning. As stated in the submitted paper (Table 11 and Section 5), all methods share the per-dataset optimizer setting, selected by a Weights & Biases sweep on FedAvg (32 trials, seed 42), and each baseline's budget follows the same rule as MAPO's ; baseline-specific parameters were tuned with the same procedure. In the comparisons added here, every method uses MAPO's uplink, and its learning rate is tuned separately. Our response to Reviewer EsfD (W1 and Q1) lists all grids, settings, and budgets.
(b) Clients joining or leaving. Communication is unaffected: a participating client always uploads scalars, MAPO keeps no client state, and a returning client downloads the current model as in FedAvg. Final accuracy at round 500 (3 seeds; CIFAR-10 = 41,412, FMNIST = 416):
| Setting (clients per round) | CIFAR-10: MAPO | FedAvg | FMNIST: MAPO | FedAvg |
|---|---|---|---|---|
| 5% sampled (5) | 53.4 ± 2.5 | 48.3 ± 1.8 | 80.8 ± 1.3 | 86.0 ± 2.3 |
| 10% sampled (10) | 57.2 ± 2.5 | 54.8 ± 3.2 | 84.4 ± 0.4 | 87.2 ± 2.1 |
| 20% sampled (20) | 59.1 ± 3.4 | 57.8 ± 2.6 | 85.6 ± 0.9 | 86.2 ± 1.5 |
| available w.p. 0.5 (5) | 55.8 ± 4.5 | 49.6 ± 8.3 | 81.2 ± 0.2 | 81.9 ± 1.1 |
| available w.p. 0.2 (2) | 42.2 ± 6.0 | 14.7 ± 8.2 | 67.5 ± 3.1 | 29.0 ± 32.9 |
| half join after round 250 | 58.5 ± 2.8 | 55.5 ± 3.2 | 84.3 ± 1.0 | 86.3 ± 1.2 |
On CIFAR-10, MAPO ends above uncompressed FedAvg in every setting by 3.6% of its uplink, and under severe churn it keeps training on both datasets while FedAvg collapses.
(c) Local iterations. Yes, the advantage holds. CIFAR-10, = 41,412, 3 seeds, final accuracy:
| Method | 1 epoch | 2 epochs | 5 epochs | 10 epochs |
|---|---|---|---|---|
| MAPO (3.6% uplink) | 47.9 ± 0.8 | 53.5 ± 2.5 | 57.2 ± 2.5 | 54.7 ± 1.4 |
| Model-wide PowerSGD (3.6%) | 33.6 ± 1.0 | 37.3 ± 1.7 | 49.5 ± 1.8 | 53.8 ± 0.8 |
| FedAvg (100%) | 42.2 ± 2.4 | 49.4 ± 2.9 | 54.8 ± 3.2 | 53.6 ± 0.5 |
Averaging the coefficients stays exact for any number of local steps, because all clients share the basis.
In the revised paper, Appendix E.2 gives the tuning protocol, the new Appendix F.1 (Table 19) gives the join-and-leave results, and the new Appendix G.3 (Table 25) gives the local-epoch study.
W3: Scope of the contributions.
Response to W3:
We agree that the scope should be clear. We do not claim that random subspaces [2], seed-generated factors [3, 4, 5], or redrawn subspaces [6] are new by themselves; the submitted paper already cites Li et al. (2018), Aghajanyan et al. (2021), and Konečný et al. (2016). If the reviewer has a specific work in mind, we would appreciate a reference.
What MAPO paper adds:
- A model-wide, shape-free parameterization with any integer budget and no layer-shape floors (W1).
- An exact, essentially free per-round refresh: all clients regenerate the same basis from one seed, so averaging coefficients is exactly averaging updates, at below 0.1% of local compute up to Llama-3.2-3B.
- Local training inside the subspace: with the same seed and uplink, it beats projecting a full local update afterward by 10.8–26.1 points on CIFAR-10 (Table 24).
- Analysis: Lemma 3.2, Props. 3.3–3.5, Thm. 4.5, and the frozen-basis floor of Prop. I.10.
A FedKSeed-style baseline [3] at MAPO's uplink reaches 88.3 / 72.2 / 14.2 on MNIST / FMNIST / CIFAR-10 vs 98.0 / 85.4 / 46.4 for MAPO (Appendix G.5); Section 2 and the new Appendix K discuss the related work.
We thank the reviewer again. The new comparisons clarify the benefit of avoiding layer-by-layer rank selection: MAPO supports any integer coefficient budget without natural-shape rank floors, while achieving similar observed accuracy to the segmented layer-wise MAPO variant. The revision also defines the contribution more precisely relative to prior work.
We would be grateful if the reviewer would consider updating their score in light of these comparisons and clarifications. If the reviewer has a specific prior work in mind for W3, we would appreciate the reference so that we can discuss the relationship directly.
[1] Y.-D. Kim et al. Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications. ICLR, 2016.
[2] F. Gressmann et al. Improving Neural Network Training in Low Dimensional Random Bases. NeurIPS, 2020.
[3] Z. Qin et al. Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes. ICML, 2024.
[4] P. Nooralinejad et al. PRANC: Pseudo RAndom Networks for Compacting Deep Models. ICCV, 2023.
[5] S. A. Koohpayegani et al. NOLA: Compressing LoRA using Linear Combination of Random Basis. ICLR, 2024.
[6] L. Melas-Kyriazi and F. Wang. Intrinsic Gradient Compression for Scalable and Efficient Federated Learning. FL4NLP Workshop (ACL), 2022.
Follow-up on our responses
Dear Reviewer vYix,
Thank you for your time and careful assessment of our work. Following your question, we compared MAPO with layer-wise PowerSGD, Tucker-2, and two layer-wise MAPO variants. The decompositions cannot run at Table 2's MNIST/FMNIST budget (their rank-one floors are 7.1–16.6× MAPO's k = 128). At those floors on FMNIST, MAPO is significantly better than PowerSGD (p = 0.001 / 0.046) and than Tucker-2 without error feedback (p = 0.005), and segmented layer-wise MAPO matches model-wide MAPO, as our Propositions predict.
As the discussion period ends soon, we would like to check whether our responses address your concerns, and we would be happy to clarify anything further.
Paper19708 Authors
Response to Reviewer vYix (1/2)
We thank the reviewer for the careful assessment and for finding that MAPO is very concise and easy to understand, that it avoids the need to select low rank on a layer-by-layer basis, and that the paper features robust and comprehensive experiments. We answer every question below. In particular, we added:
All changes are marked in blue in the revised paper, and we provide their locations below.
Response to W1:
We agree that layer-wise decompositions can be applied to any architecture. The difference is that a layer-wise decomposition ties its factors and its minimum payload to each layer's shape, whereas MAPO uses one model-wide factorization with a single budget k. We ran all three comparisons the reviewer asked for.
(a) What the submitted paper already shows. Propositions 3.3–3.5 show that, at a budget both can reach, model-wide and layer-wise factorizations with random bases have the same expected reconstruction error, 1−k/d, and that the model-wide basis needs about half of the factor parameters. They predict that MAPO matches a layer-wise version of itself where both can run, and that only MAPO can run at small budgets. The new experiments confirm both.
(b) Small budgets: MAPO wins or is the only option. At rank 1 in every layer, layer-wise PowerSGD needs 1,077 and 10,211 and Tucker-2 [1] needs 911 and 4,095 scalars per round on the MNIST/FMNIST and CIFAR-10 CNNs, while MAPO runs at any k≥1. Results at these floors (3 seeds, best accuracy, same uplink in each row):
On FMNIST, MAPO is significantly better than PowerSGD at 1,077 (p = 0.001 / 0.046) and than Tucker-2 without EF at 911 (p = 0.005), and comparable to Tucker-2 with EF (p = 0.28); on MNIST all are within noise. At the k = 128 of Table 2, and below (e.g., 77.7 on FMNIST with 34 scalars), only MAPO can run.
(c) Larger budgets: comparable accuracy at a fraction of the cost. CIFAR-10, k = 41,412, 3 seeds, learning rate tuned per method; p: Welch test vs model-wide MAPO:
As predicted, segmented layer-wise MAPO matches model-wide MAPO. No decomposition is significantly better: Tucker-2 is within noise but needs at least 7.6× MAPO's compression compute and, with EF, a full-model residual on every client.
(d) Why model-wide MAPO. It is the only method that runs at any budget, is never significantly worse and is better at small budgets, keeps no client state, and has one hyperparameter k with the same code for every architecture.
In the revised paper, the new Appendix D.2 contains this comparison: Table 11 gives the minimum payloads, and Table 12 the accuracies at the floors and at matched uplink.