acceptodds
Under review as a conference paper at ICLR 2027

Model-Agnostic Projection Optimization for Communication-Efficient Federated Learning

Abstract

Federated learning enables collaborative model training across distributed clients without sharing sensitive data. However, communication overhead remains a significant bottleneck. Low-rank decomposition techniques address this by approximating each layer’s weights or gradients as a product of low-rank matrices, thereby reducing communication costs in FL. While effective, these methods are typically layer-wise and constrained by the layer's architecture and shapes, limiting their flexibility and performance. We propose **Model-Agnostic Projection Optimization** (MAPO), a novel model-wide low-rank parametrization method that factorizes the full model gradient into a *fixed reconstruction matrix* and a *trainable projection vector*, thereby avoiding layer-wise decomposition and architecture constraints. MAPO directly optimizes the projection in a randomly sampled subspace, with all clients generating the reconstruction matrix via a shared random seed, incurring no additional communication overhead. By decoupling the gradient from architecture through reshaping and enabling communication-free exploration of subspaces via seed sharing, MAPO provides a flexible and efficient low-rank representation. Across seven FL benchmarks, MAPO reaches 92–99.7% of FedAvg's accuracy while using 0.13–3.1% of its uplink, exceeding all evaluated baselines.

Primary area:
optimization
Keywords:
Federated Learning, Low-rank Adaptation, Communication efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vYixOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MAPO, which flattens the entire model gradient and reshapes it into a matrix (model-independence), and performs parametric updates using a randomly regenerated reconstruction vector and a trainable projection vector in each iteration. The client performs optimization within a randomly sampled subspace and uploads only the random seed, thereby achieving communication efficiency.

Strengths

The method is very concise and easy to understand, and it avoids the need to select low rank on a layer-by-layer basis. Furthermore, the paper features robust and comprehensive experiments, and the results demonstrate the effectiveness of MAPO.

Weaknesses

  1. The paper points out the drawbacks of layer-by-layer rank selection and emphasizes model-independence. In fact, for both low-rank matrix decomposition and low-rank tensor decomposition, each layer of most models can be viewed as a matrix or tensor. Therefore, such methods are also model-agnostic. It would be helpful if the authors could include a comparison with low-rank matrix decomposition and low-rank tensor decomposition methods (which are layer-by-layer decomposition approaches). Additionally, the authors should compare MAPO with a layer-by-layer decomposition version of MAPO to better demonstrate its effectiveness.
  2. Are the other baselines in the experimental section the result of thorough hyperparameter tuning? How were the hyperparameters for the comparison baselines selected? Furthermore, how do accuracy and communication overhead behave when clients join or leave the system? If the number of local training iterations per client is altered, would MAPO’s advantages still hold?
  3. The concept of MAPO was not first proposed in this paper. Therefore, I suggest that the authors clearly define the scope of their contributions.
Critical strengths

The method is very concise and easy to understand, and it avoids the need to select low rank on a layer-by-layer basis. Furthermore, the paper features robust and comprehensive experiments, and the results demonstrate the effectiveness of MAPO.

Critical weaknesses
  1. The paper points out the drawbacks of layer-by-layer rank selection and emphasizes model-independence. In fact, for both low-rank matrix decomposition and low-rank tensor decomposition, each layer of most models can be viewed as a matrix or tensor. Therefore, such methods are also model-agnostic. It would be helpful if the authors could include a comparison with low-rank matrix decomposition and low-rank tensor decomposition methods (which are layer-by-layer decomposition approaches). Additionally, the authors should compare MAPO with a layer-by-layer decomposition version of MAPO to better demonstrate its effectiveness.
  2. Are the other baselines in the experimental section the result of thorough hyperparameter tuning? How were the hyperparameters for the comparison baselines selected? Furthermore, how do accuracy and communication overhead behave when clients join or leave the system? If the number of local training iterations per client is altered, would MAPO’s advantages still hold?
  3. The concept of MAPO was not first proposed in this paper. Therefore, I suggest that the authors clearly define the scope of their contributions.
Questions for authors
  1. The paper points out the drawbacks of layer-by-layer rank selection and emphasizes model-independence. In fact, for both low-rank matrix decomposition and low-rank tensor decomposition, each layer of most models can be viewed as a matrix or tensor. Therefore, such methods are also model-agnostic. It would be helpful if the authors could include a comparison with low-rank matrix decomposition and low-rank tensor decomposition methods (which are layer-by-layer decomposition approaches). Additionally, the authors should compare MAPO with a layer-by-layer decomposition version of MAPO to better demonstrate its effectiveness.
  2. Are the other baselines in the experimental section the result of thorough hyperparameter tuning? How were the hyperparameters for the comparison baselines selected? Furthermore, how do accuracy and communication overhead behave when clients join or leave the system? If the number of local training iterations per client is altered, would MAPO’s advantages still hold?
  3. The concept of MAPO was not first proposed in this paper. Therefore, I suggest that the authors clearly define the scope of their contributions.
5 comments
AuthorsOpenReview ↗

Response to Reviewer vYix (1/2)

We thank the reviewer for the careful assessment and for finding that MAPO is very concise and easy to understand, that it avoids the need to select low rank on a layer-by-layer basis, and that the paper features robust and comprehensive experiments. We answer every question below. In particular, we added:

  • a comparison with layer-wise low-rank matrix and tensor decompositions, and with two layer-wise versions of MAPO (W1);
  • the tuning protocol of every baseline, results when clients join or leave, and for 1 to 10 local epochs (W2);
  • a clear statement of the scope of our contributions (W3).

All changes are marked in blue in the revised paper, and we provide their locations below.


W1: Comparison with layer-wise low-rank decompositions, and with a layer-wise MAPO.

Response to W1:

We agree that layer-wise decompositions can be applied to any architecture. The difference is that a layer-wise decomposition ties its factors and its minimum payload to each layer's shape, whereas MAPO uses one model-wide factorization with a single budget kk. We ran all three comparisons the reviewer asked for.

(a) What the submitted paper already shows. Propositions 3.3–3.5 show that, at a budget both can reach, model-wide and layer-wise factorizations with random bases have the same expected reconstruction error, 1−k/d1-k/d, and that the model-wide basis needs about half of the factor parameters. They predict that MAPO matches a layer-wise version of itself where both can run, and that only MAPO can run at small budgets. The new experiments confirm both.


(b) Small budgets: MAPO wins or is the only option. At rank 1 in every layer, layer-wise PowerSGD needs 1,077 and 10,211 and Tucker-2 [1] needs 911 and 4,095 scalars per round on the MNIST/FMNIST and CIFAR-10 CNNs, while MAPO runs at any k≥1k \ge 1. Results at these floors (3 seeds, best accuracy, same uplink in each row):

Dataset, uplinkMAPOLayer-wise PowerSGD, with / without EFLayer-wise Tucker-2, with / without EF
MNIST, 42097.9 ± 0.1infeasibleinfeasible
MNIST, 91198.1 ± 0.0infeasible98.2 ± 0.2 / 96.7 ± 0.3
MNIST, 1,07797.3 ± 1.297.3 ± 0.4 / 97.7 ± 0.1—
FMNIST, 41685.4 ± 0.1infeasibleinfeasible
FMNIST, 91186.0 ± 0.8infeasible85.0 ± 1.1 / 81.2 ± 1.1
FMNIST, 1,07785.9 ± 0.681.6 ± 0.7 / 83.9 ± 1.0—

On FMNIST, MAPO is significantly better than PowerSGD at 1,077 (pp = 0.001 / 0.046) and than Tucker-2 without EF at 911 (pp = 0.005), and comparable to Tucker-2 with EF (pp = 0.28); on MNIST all are within noise. At the kk = 128 of Table 2, and below (e.g., 77.7 on FMNIST with 34 scalars), only MAPO can run.


(c) Larger budgets: comparable accuracy at a fraction of the cost. CIFAR-10, kk = 41,412, 3 seeds, learning rate tuned per method; pp: Welch test vs model-wide MAPO:

MethodExtra client stateBest accuracypp
Layer-wise PowerSGD, with / without EFfull model / none49.4 ± 0.6 / 56.7 ± 0.80.01 / 0.13
Layer-wise Tucker-2, with / without EFfull model / none63.2 ± 0.9 / 62.1 ± 0.60.10 / 0.20
Layer-wise MAPO, natural layer shapesnone55.9 ± 1.10.08
Layer-wise MAPO, segmentednone60.8 ± 0.90.53
MAPO (model-wide)none59.7 ± 2.2—

As predicted, segmented layer-wise MAPO matches model-wide MAPO. No decomposition is significantly better: Tucker-2 is within noise but needs at least 7.6× MAPO's compression compute and, with EF, a full-model residual on every client.


(d) Why model-wide MAPO. It is the only method that runs at any budget, is never significantly worse and is better at small budgets, keeps no client state, and has one hyperparameter kk with the same code for every architecture.

In the revised paper, the new Appendix D.2 contains this comparison: Table 11 gives the minimum payloads, and Table 12 the accuracies at the floors and at matched uplink.

AuthorsOpenReview ↗

Response to Reviewer vYix (2/2)

W2: Tuning of the baselines, clients joining or leaving, and local iterations.

Response to W2:

(a) Tuning. As stated in the submitted paper (Table 11 and Section 5), all methods share the per-dataset optimizer setting, selected by a Weights & Biases sweep on FedAvg (32 trials, seed 42), and each baseline's budget follows the same rule as MAPO's kk; baseline-specific parameters were tuned with the same procedure. In the comparisons added here, every method uses MAPO's uplink, and its learning rate is tuned separately. Our response to Reviewer EsfD (W1 and Q1) lists all grids, settings, and budgets.


(b) Clients joining or leaving. Communication is unaffected: a participating client always uploads kk scalars, MAPO keeps no client state, and a returning client downloads the current model as in FedAvg. Final accuracy at round 500 (3 seeds; CIFAR-10 kk = 41,412, FMNIST kk = 416):

Setting (clients per round)CIFAR-10: MAPOFedAvgFMNIST: MAPOFedAvg
5% sampled (5)53.4 ± 2.548.3 ± 1.880.8 ± 1.386.0 ± 2.3
10% sampled (10)57.2 ± 2.554.8 ± 3.284.4 ± 0.487.2 ± 2.1
20% sampled (20)59.1 ± 3.457.8 ± 2.685.6 ± 0.986.2 ± 1.5
available w.p. 0.5 (5)55.8 ± 4.549.6 ± 8.381.2 ± 0.281.9 ± 1.1
available w.p. 0.2 (2)42.2 ± 6.014.7 ± 8.267.5 ± 3.129.0 ± 32.9
half join after round 25058.5 ± 2.855.5 ± 3.284.3 ± 1.086.3 ± 1.2

On CIFAR-10, MAPO ends above uncompressed FedAvg in every setting by 3.6% of its uplink, and under severe churn it keeps training on both datasets while FedAvg collapses.


(c) Local iterations. Yes, the advantage holds. CIFAR-10, kk = 41,412, 3 seeds, final accuracy:

Method1 epoch2 epochs5 epochs10 epochs
MAPO (3.6% uplink)47.9 ± 0.853.5 ± 2.557.2 ± 2.554.7 ± 1.4
Model-wide PowerSGD (3.6%)33.6 ± 1.037.3 ± 1.749.5 ± 1.853.8 ± 0.8
FedAvg (100%)42.2 ± 2.449.4 ± 2.954.8 ± 3.253.6 ± 0.5

Averaging the coefficients stays exact for any number of local steps, because all clients share the basis.

In the revised paper, Appendix E.2 gives the tuning protocol, the new Appendix F.1 (Table 19) gives the join-and-leave results, and the new Appendix G.3 (Table 25) gives the local-epoch study.


W3: Scope of the contributions.

Response to W3:

We agree that the scope should be clear. We do not claim that random subspaces [2], seed-generated factors [3, 4, 5], or redrawn subspaces [6] are new by themselves; the submitted paper already cites Li et al. (2018), Aghajanyan et al. (2021), and Konečný et al. (2016). If the reviewer has a specific work in mind, we would appreciate a reference.

What MAPO paper adds:

  1. A model-wide, shape-free parameterization with any integer budget kk and no layer-shape floors (W1).
  2. An exact, essentially free per-round refresh: all clients regenerate the same basis from one seed, so averaging coefficients is exactly averaging updates, at below 0.1% of local compute up to Llama-3.2-3B.
  3. Local training inside the subspace: with the same seed and uplink, it beats projecting a full local update afterward by 10.8–26.1 points on CIFAR-10 (Table 24).
  4. Analysis: Lemma 3.2, Props. 3.3–3.5, Thm. 4.5, and the frozen-basis floor of Prop. I.10.

A FedKSeed-style baseline [3] at MAPO's uplink reaches 88.3 / 72.2 / 14.2 on MNIST / FMNIST / CIFAR-10 vs 98.0 / 85.4 / 46.4 for MAPO (Appendix G.5); Section 2 and the new Appendix K discuss the related work.


We thank the reviewer again. The new comparisons clarify the benefit of avoiding layer-by-layer rank selection: MAPO supports any integer coefficient budget without natural-shape rank floors, while achieving similar observed accuracy to the segmented layer-wise MAPO variant. The revision also defines the contribution more precisely relative to prior work.

We would be grateful if the reviewer would consider updating their score in light of these comparisons and clarifications. If the reviewer has a specific prior work in mind for W3, we would appreciate the reference so that we can discuss the relationship directly.

[1] Y.-D. Kim et al. Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications. ICLR, 2016.
[2] F. Gressmann et al. Improving Neural Network Training in Low Dimensional Random Bases. NeurIPS, 2020.
[3] Z. Qin et al. Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes. ICML, 2024.
[4] P. Nooralinejad et al. PRANC: Pseudo RAndom Networks for Compacting Deep Models. ICCV, 2023.
[5] S. A. Koohpayegani et al. NOLA: Compressing LoRA using Linear Combination of Random Basis. ICLR, 2024.
[6] L. Melas-Kyriazi and F. Wang. Intrinsic Gradient Compression for Scalable and Efficient Federated Learning. FL4NLP Workshop (ACL), 2022.
AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer vYix,

Thank you for your time and careful assessment of our work. Following your question, we compared MAPO with layer-wise PowerSGD, Tucker-2, and two layer-wise MAPO variants. The decompositions cannot run at Table 2's MNIST/FMNIST budget (their rank-one floors are 7.1–16.6× MAPO's k = 128). At those floors on FMNIST, MAPO is significantly better than PowerSGD (p = 0.001 / 0.046) and than Tucker-2 without error feedback (p = 0.005), and segmented layer-wise MAPO matches model-wide MAPO, as our Propositions predict.

As the discussion period ends soon, we would like to check whether our responses address your concerns, and we would be happy to clarify anything further.

Paper19708 Authors

Reviewer_ZNf8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Model-Agnostic Projection Optimization (MAPO), a communication-efficient federated learning method that represents full-model updates using a shared random reconstruction vector and a trainable projection vector, with the random vector refreshed each communication round. The authors provide theoretical analyses and evaluate MAPO across seven federated learning benchmarks, demonstrating competitive accuracy with substantially reduced communication overhead.

Critical strengths
  1. The paper presents an intuitive motivation and a clear, easy-to-follow methodological framework.

  2. The paper provides theoretical analyses and demonstrates promising accuracy and communication efficiency across multiple federated learning benchmarks.

Critical weaknesses
  1. Prior work, such as "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning" (Aghajanyan et al., 2021), has already explored model-wide low-dimensional optimization without relying on individual layer structures or shapes. Although MAPO cites this work and introduces a different projection structure and dynamic subspace exploration, the existing ablation studies mainly demonstrate the benefits of refreshing the reconstruction vector, without sufficiently isolating the contribution of MAPO's specific parameterization. A more controlled comparison with existing model-wide low-dimensional optimization methods, such as a federated adaptation of Aghajanyan et al. (2021), would help clarify its independent advantages.

  2. The communication savings reported in Table 2 depend on a target determined by the worst-performing baseline, which limits their interpretation beyond that particular accuracy level. Although Figures 5 and 8 provide communication–accuracy curves, Table 7 evaluates communication at 80% and 90% of FedAvg’s final accuracy, but only on SVHN and CIFAR-10 with a limited set of methods. Extending this protocol to the benchmarks and baselines in Table 2 and presenting the results in a table, with unattained targets marked as “Not reached,” would provide a clearer comparison of communication savings across multiple, practically relevant accuracy targets.

Questions for authors
  1. Prior work, such as "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning" (Aghajanyan et al., 2021), has already explored model-wide low-dimensional optimization without relying on individual layer structures or shapes. Although MAPO cites this work and introduces a different projection structure and dynamic subspace exploration, the existing ablation studies mainly demonstrate the benefits of refreshing the reconstruction vector, without sufficiently isolating the contribution of MAPO's specific parameterization. A more controlled comparison with existing model-wide low-dimensional optimization methods, such as a federated adaptation of Aghajanyan et al. (2021), would help clarify its independent advantages.

  2. The communication savings reported in Table 2 depend on a target determined by the worst-performing baseline, which limits their interpretation beyond that particular accuracy level. Although Figures 5 and 8 provide communication–accuracy curves, Table 7 evaluates communication at 80% and 90% of FedAvg’s final accuracy, but only on SVHN and CIFAR-10 with a limited set of methods. Extending this protocol to the benchmarks and baselines in Table 2 and presenting the results in a table, with unattained targets marked as “Not reached,” would provide a clearer comparison of communication savings across multiple, practically relevant accuracy targets.

  3. Table 7 reports per-round communication costs of 0.78 MB, 6.25 MB, and 0.32 GB for MAPO-2k, MAPO-16k, and MAPO-40k, respectively. Could the authors clarify how “Com/Round” is defined, including whether it aggregates traffic across clients or multiple synchronization events? Under Algorithm 1, each participating client uploads one k-dimensional vector after completing its local updates. What accounts for the disproportionate increase from MAPO-16k to MAPO-40k?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZNf8 (1/2)

We thank the reviewer for the careful assessment and for recognizing the paper's intuitive motivation, clear methodological framework, theoretical analyses, and promising communication efficiency. We answer every question below. In particular, we added:

  • a federated adaptation of Aghajanyan et al. (2021) that separates projection structure from subspace refresh, on three CNNs and RoBERTa-base, with the compute, memory, and time of each projection (W1 and Q1);
  • communication to 80%, 90%, and 95% of FedAvg's accuracy for every Table 2 method (W2 and Q2);
  • the corrected MAPO-40k entry and its formula (Q3).

All changes are marked in blue in the revised paper.


W1 and Q1: A controlled comparison with a federated adaptation of Aghajanyan et al. (2021).

Response to W1 and Q1 (1/2):

The submitted paper cites Aghajanyan et al. (2021), and its ablations isolate the refresh (fresh, frozen, and trainable bases), but not the projection structure. We ran the federated adaptation the reviewer suggested and varied the two ingredients independently, point by point. The results show that MAPO's accuracy advantage comes from refreshing the subspace, which other projections can also benefit from; however, MAPO's factorization costs less compute and far less memory, which makes the refresh feasible for FL with resource-constrained clients.

(a) Design. We cross two projections, Fastfood (the federated intrinsic-dimension baseline) and MAPO's segmented rank-one basis, with two subspace rules, fixed for all rounds or refreshed every round from a shared seed, and add fixed Fastfood with the learned scales of SAID and a dense Gaussian projection. Every arm uploads the same kk coefficients, and all arms share clients, partition, training horizon, and local work; learning rates are tuned separately per arm on seed 42 over the same grid and then run on seeds 43 and 44. Budgets: kk = 420 / 416 / 41,412 on MNIST / FMNIST / CIFAR-10 (about 3.7% of the model).


(b) Results (best test accuracy, 3 seeds):

ProjectionSubspaceMNISTFMNISTCIFAR-10
Fastfood (fed. Aghajanyan)fixed95.0±0.282.4±0.357.8±1.0
Fastfood + SAIDfixed95.5±0.182.4±0.757.0±0.6
MAPO basisfixed95.1±0.781.3±0.358.2±0.4
Fastfoodrefreshed98.5±0.086.5±0.762.4±1.4
MAPOrefreshed97.9±0.185.4±0.159.7±2.2

At tighter budgets the gap to a fixed subspace widens:

MethodFMNIST, 1%FMNIST, 0.3%FMNIST, 0.1%CIFAR-10, 1%
Fastfood, fixed (fed. Aghajanyan)72.5±0.356.2±2.832.0±2.454.6±1.2
MAPO basis, fixed73.4±0.353.2±2.031.8±4.454.5±1.0
Fastfood, refreshed83.8±0.477.3±0.669.5±0.260.2±1.5
MAPO82.7±0.377.7±0.463.4±3.659.1±2.0

cc Pretrained language model. The setting closest to Aghajanyan et al.: RoBERTa-base on SST-2, every subspace arm sending 12,465 scalars per round (0.01% of the parameters), 100 IID clients, 10 per round, one local epoch, 50 rounds, learning rates tuned per arm:

MAPOFastfood, refreshedFastfood, fixedFastfood + SAIDFedAvg (100% uplink)
93.4±0.393.4±0.392.8±0.292.9±0.194.8±0.2

(d) What this isolates.

  1. A federated Aghajanyan baseline loses clearly to MAPO on the CNNs: by 2–3 points at 3.7% of the model and by 10–31 points on FMNIST at tighter budgets; SAID does not close the gap. On pretrained RoBERTa-base, where the intrinsic dimension is low, all arms are comparable.
  2. Refreshing the subspace is what drives accuracy: with a fixed subspace, all projections are within 1.3 points of each other, and every projection gains when the subspace is refreshed (for MAPO's basis, by 9.3 / 24.5 / 31.6 points on FMNIST at 1% / 0.3% / 0.1%), as Prop. I.10 predicts.
  3. With refresh, Fastfood is somewhat more accurate on these benchmarks (0.6–2.7 points at k/d ≈ 3.7%), but MAPO’s parameterization is what makes the refresh cheap at any model size (next comment).

(e) The independent contribution. Model-wide intrinsic-dimension training predates MAPO, as the reviewer notes, and intrinsic gradient compression [1] uses shared random projections in FL. MAPO contributes the segmented parameterization with its tiny seeded basis, which makes a per-round refresh essentially free at any scale and allows any integer budget kk; local training directly in coefficient space; and the analysis: convergence for every kk (Thm. 4.5) and the stationarity floor of a frozen subspace (Prop. I.10), now with a remark extending it to any frozen linear projection.


The cost of the refresh (f) follows in the next comment.


[1] L. Melas-Kyriazi and F. Wang. [Intrinsic Gradient Compression for Scalable and Efficient Federated Learning](https://aclanthology.org/2022.fl4nlp-1.4/). FL4NLP Workshop (ACL), 2022.
AuthorsOpenReview ↗

Response to Reviewer ZNf8 (2/2)

Response to W1 and Q1 (2/2):

(f) Cost of a refreshed basis. Let dd be the model size and kk the number of uploaded coefficients. MAPO stores one seeded vector of length ⌈d/k⌉\lceil d/k\rceil and applies it to all kk segments, so applying the basis and its transpose costs O(d)O(d) per local step. Fastfood uses a transform of length DD, the next power of two ≥d\ge d, with O(Dlog⁡D)O(D\log D) work and O(D)O(D) memory, and a stored dense projection needs O(dk)O(dk) work and memory. Extra arithmetic per round, as % of FedAvg's local training compute, and basis memory:

Model and budgetMAPO computeFastfood computeMAPO basis (fp32)Fastfood buffers (fp32)
MNIST CNN, kk = 4200.04%0.86%108 B0.13–0.39 MB
CIFAR-10 CNN, kk = 41,4120.06%2.6%112 B17–50 MB
ViT-B/16, k=220k = 2^{20}0.04%1.8%328 B1.1–3.2 GB
RoBERTa-large, k≈d/1,000k \approx d/1{,}0000.004%0.23%about 4 KB4.3–12.9 GB
Llama-3.2-3B, k=225k = 2^{25}0.10%5.8%384 B34–103 GB

Measured on one RTX 3090 at k≈d/100k \approx d/100, one reconstruction plus projection takes 0.08 ms for MAPO vs 1.32 ms for Fastfood on the CIFAR-10 CNN, and 2.07 ms vs 115 ms on ViT-B/16; our Fastfood implementation runs out of memory at RoBERTa-large scale.

In the revised paper, Table 21 (new Appendix G.2) reports all three comparisons, and Appendix E.2 reports their tuning protocol. Tables 22 and 23 (Appendix G.2) give the measured basis costs and the compute and memory analysis. Section 2 and the new Appendix K discuss the relation to intrinsic-dimension methods; and the remark after Proposition I.10 covers any frozen linear projection.


W2 and Q2: Communication at several accuracy targets.

Response to W2 and Q2:

The submitted Table 2 reports communication to the worst baseline's accuracy, and the submitted Table 7 (Table 8 in the revision) evaluates 80% and 90% of FedAvg on SVHN and CIFAR-10 only; the bottom rows of Figures 5 and 8 show the communication needed for any accuracy level. Following the reviewer's suggestion, the new Table 4 extends this to every Table 2 method on five benchmarks. Each entry is the method's cumulative uplink to first reach the target (a fraction of FedAvg's best accuracy), as % of FedAvg's; a target counts as reached only if the method's mean maximum accuracy in Table 2 reaches it.

TargetMethodMNISTFMNISTCIFAR-10ShakespeareSent140
80%Quantize9.07.513.012.411.9
Sparse4.44.6N/R2.62.2
EvoFed13.34.3N/R0.470.71
FedLoRU13.024.8N/RN/R0.31
MAPO1.81.31.30.120.35
90%Quantize12.06.911.3N/R16.2
Sparse10.312.8N/RN/R1.9
EvoFed8.313.0N/RN/R0.55
FedLoRU40.2N/RN/RN/RN/R
MAPO2.52.31.30.140.40
95%Quantize10.75.811.1N/R22.6
SparseN/RN/RN/RN/R1.6
EvoFed8.1N/RN/RN/RN/R
FedLoRUN/RN/RN/RN/RN/R
MAPO3.72.21.40.170.28

MAPO needs the least uplink among the compressed methods at every target on every benchmark except Sent140 at 80%, where FedLoRU reaches the target with slightly less uplink (0.31% vs 0.35%).

In the revised paper, the new Table 4 (Appendix B) gives these values.


Q3: How Com/Round is defined, and MAPO-40k

Response to Q3:

Each participating client uploads its kk coefficients once per round, after local training, as in Algorithm 1; the reconstruction vector is regenerated from the seed and never uploaded, and no other synchronization is counted. Com/Round sums these uploads over the 100 clients that upload in a round, Com/Round=100×k×4\text{Com/Round} = 100 \times k \times 4 bytes:

ConfigurationkkUplink per clientCom/Round
MAPO-2k2,0488 KB0.78 MB
MAPO-16k16,38464 KB6.25 MB
MAPO-40k40,960160 KB15.6 MB

The MAPO-40k entry was a computation error: it should read 15.6 MB (not 0.32 GB), exactly 2.5× the 16k entry, proportional to kk. Its cumulative costs are corrected accordingly; e.g., 12 rounds to the 80% target on SVHN (IID) take about 0.18 GB instead of 3.84 GB.

In the revised paper, Table 8 (Appendix D) (Table 7 in the submission) has the corrected entries, and its caption provides the Com/Round formula.


We appreciate the reviewer’s suggestion to include a federated Aghajanyan comparison. It clarifies that the CNN accuracy gains come from refreshing the subspace, while MAPO’s contribution is making this refresh inexpensive: below 0.1% of local compute in the evaluated models up to Llama-3.2-3B. The fixed-subspace baseline trails MAPO by 2–31 points on the CNNs. Together with the multi-target communication results in Table 4 and the corrected Com/Round entry in Table 8, these additions address W2 and Q3 and clarify MAPO’s advantages and scope.

If this evidence resolves the reviewer’s concerns, we would be grateful if they would reconsider their score. We welcome any remaining questions.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer ZNf8,

Thank you for your time and careful assessment of our work. Following your question, we added a federated adaptation of Aghajanyan et al.: a fixed intrinsic-dimension subspace is 2–3 points below MAPO at k/d ≈ 3.7% and 4.5–31.4 points below at k/d = 1–0.1%.

Also, we showed that refreshing the subspace is what matters, and MAPO makes the refresh cheap: 0.08 ms vs 1.32 ms per projection on the CIFAR-10 CNN and 2.07 ms vs 115 ms on ViT-B/16.

As the discussion period ends soon, we would like to check whether our responses address your concerns, and we would be happy to clarify anything further.

Paper19708 Authors

Reviewer_EsfDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Model-Agnostic Projection Optimization (MAPO), a communication-efficient federated-learning method that flattens the full model update, reshapes it into k segments, and parameterizes each segment using one coefficient and a shared random reconstruction vector. A fresh reconstruction vector is regenerated from a common seed each round, while clients optimize and upload only k coefficients. Since all clients use the same random vector, coefficient averaging exactly equals averaging the reconstructed updates. The paper analyzes expected reconstruction error, model-wide versus layer-wise factorization size, fresh versus frozen subspaces, and nonconvex convergence for a one-local-step SGD specialization. Experiments with seven federated benchmarks, GLUE fine-tuning, ViT and Llama models, several compression and low-rank baselines, communication-budget sweeps, and fresh/frozen/trainable-basis ablations. The claimed result is near-FedAvg accuracy at a small fraction of its cumulative uplink.

Critical strengths

Elegant, architecture-independent construction. Sections 3–3.1 turn the whole update into a single segmented rank-one parameterization. Sharing only a seed for A_t makes aggregation exact and allows any integer coefficient budget k, avoiding layer-shape rank floors.

Clear theoretical intuition. Lemma 3.2 and Propositions 3.3–3.5 characterize expected projection error and communication matching; Theorem 4.5 shows that fresh Gaussian subspaces yield a scaled-unbiased update with a second-moment penalty depending on n=\lceil d/k\rceil. Proposition I.10 cleanly explains why a frozen basis can create a stationarity floor.

Substantial empirical breadth. Table 2 covers seven datasets and multiple architectures; Appendices C–D add GLUE, ViT, Llama, Factorized-FL, and a matched-uplink GaLore-FL implementation. Table 12 additionally studies severe label skew.

Critical weaknesses

Baseline tuning and budget matching are insufficiently transparent. Section E.1 explains MAPO's k search, but the exact search spaces, selected budgets, stopping/checkpoint rules, and number of trials for Sparse, Quantize, EvoFed, and FedLoRU are not tabulated.

Statistical support is modest relative to the breadth of claims. Main results use only three runs (Table 2), and the large-model, GLUE, GaLore-FL, extreme-skew, and communication-threshold tables generally omit uncertainty or significance. The 128-example GSM8K subset in Section C.1 is especially small. More seeds or confidence intervals on the key matched-budget comparisons are needed.

Questions for authors

What exact search spaces and selection criteria were used for every baseline's compression budget and optimizer hyperparameters?

Can confidence intervals or additional seeds be provided for the matched-uplink GaLore-FL comparison and the ViT/Llama experiments?

4 comments
AuthorsOpenReview ↗

Response to Reviewer EsfD (1/2)

We thank the reviewer for the positive assessment and for recognizing MAPO's elegant, architecture-independent construction, clear theoretical intuition, and substantial empirical breadth. We answer every question below. In particular, we added:

  • the search spaces, selected optimizer settings and budgets, stopping rule, and number of trials (W1 and Q1);
  • three-seed results for the matched-uplink GaLore-FL comparison and ViT-B/16, and 95% intervals for Llama (W2 and Q2);
  • uncertainty estimates for GLUE, label skew, and the communication thresholds, and a clarification of the GSM8K evaluation set (W2).

All changes are marked in blue in the revised paper, and their locations are given below.


W1 and Q1: Baseline tuning, budget selection, stopping rules and number of trials.

Response to W1 and Q1:

The submitted paper already reported part of this protocol: Table 11 (Table 15 in the revision) lists the training hyperparameters of FedAvg and its variants; Section 5 and Appendix E.1 give the budget rule and state that it is applied to every baseline; and Table 2 reports three runs. The revision adds the missing details, point by point:

(a) Search spaces and selection criteria (Q1). For each dataset, the optimizer setting was selected with a Bayesian sweep in Weights & Biases on FedAvg (32 trials per dataset, seed 42, objective: global test accuracy) over the grids below; all methods use the selected setting. Baseline-specific parameters, such as EvoFed's perturbation scale, were tuned separately with the same procedure. L1 and L2 were then adjusted by hand per dataset to limit FedAvg's overfitting (e.g., the CIFAR-100 weight decay was raised from 1e−31e-3 to 3e−33e-3).

HyperparameterGrid
Learning rate0.001, 0.003, 0.01, 0.02, 0.03, 0.05, 0.1, 0.2
Momentum0, 0.4, 0.8, 0.9
L1 regularization0, 5e-6, 1e-5, 1e-4
L2 regularization0, 1e-5, 5e-5, 1e-4, 1e-3

(b) Selected optimizer settings (Table 15; batch size 32 for all):

DatasetOptimizerLRMomentum / β1\beta_1L1L2 / weight decay
MNIST, FMNISTSGD0.20.900
CIFAR-10SGD0.030.41e-41e-5
CIFAR-100AdamW0.10.903e-3
TinyImageNetAdamW0.20.91e-51e-4
ShakespeareSGD0.20.95e-65e-5
Sentiment140SGD0.0010.900

(c) Selected budgets of every baseline (W1). Each method's budget follows the rule of Section 5 and Appendix E.1: starting from a high compression, the budget is increased until a larger one no longer reduces the total communication needed to reach near-optimal accuracy. Per-round uplink per client (Table 16):

MNISTFMNISTCIFAR-10CIFAR-100TinyShakesp.Sent140
Rounds5005005002002001,0002,000
Sparse keep-rate0.1%0.5%0.05%0.5%1.0%1.0%1.0%
Quantize bits4444444
EvoFed (scalars)1282042,04865,53616,3841,0242,048
FedLoRU (scalars)2561281288,1928,1922,0484,096
MAPO kk12812816,3848,19232,7681,0244,096

These budgets are per round; Table 2's communication is the cumulative uplink to the common accuracy target, relative to FedAvg's, so it also reflects how many rounds each method needs. Sparse accounting includes coordinate indices, and quantization includes scales.


(d) Stopping/checkpoint rule and number of trials (W1). Every method reports its best global test accuracy within the fixed horizon above, with no early stopping. Each dataset used 32 sweep trials, and we ran the selected configuration with three seeds (Table 2: mean ± std).


In the revised paper, Appendix E.1 describes the budget selection for every baseline, Tables 15 and 16 give the selected settings, budgets, and horizons, and the new Appendix E.2 gives the search grids, number of trials, and checkpoint rule.

AuthorsOpenReview ↗

Response to Reviewer EsfD (2/2)

W2 and Q2. Seeds, confidence intervals, and the GSM8K subset.

Response to W2 and Q2:

Every table the reviewer listed now reports seeds or intervals. We follow the reviewer's order.

(a) Matched-uplink GaLore-FL (Q2). As in the submitted Table 8 (Table 9 in the revision), the point of this comparison is that MAPO trains at any budget, whereas GaLore-FL cannot go below its integer-rank floor; at the caps both can reach, MAPO is comparable or better. Three seeds confirm this. On WideResNet-16-4 (10 IID clients, full participation, batch 128, 200 rounds, lr 0.1, seeds paired), the global model's best accuracy gives MAPO − GaLore-FL (paired pp):

Cap (actual uplink MAPO / GaLore-FL)Constant stepFirst 100 roundsSame cosine schedule for both
CIFAR-100, 10% (10.00 / 10.32%)+2.2 (0.011)+5.2+2.8 (0.003)
CIFAR-100, 2% (2.00 / 2.53%)0.0 (0.945)+6.9+7.3 (0.001)
CIFAR-100, 1% (1.02 / 1.33%)−1.6 (0.188)+5.7+9.3 (<0.001)
CIFAR-10, 10%+1.7 (0.047)+3.0+1.5 (0.019)
CIFAR-10, 2%+0.4 (0.036)+4.2+3.3 (0.001)
CIFAR-10, 1%+1.0 (0.092)+6.0+4.9 (0.001)

With a constant step, MAPO is higher in 4 of 6 cells (significantly in 3), and GaLore-FL, despite more uplink at every cap, is never significantly better in these runs. MAPO also learns faster (ahead at every cap within 100 rounds), and with the cosine schedule ηt=0.1(1+cos⁡(πt/200))/2\eta_t=0.1(1+\cos(\pi t/200))/2, which our analysis suggests for a subspace redrawn every round (Cor. I.8(iii)), it leads at every cap (all p≤0.019p\le0.019). Below GaLore-FL's floor, only MAPO trains: 58.3 ± 0.3 at 0.35% and 53.9 ± 0.2 at 0.25% uplink on CIFAR-100. On ViT-B/16 (three seeds, 100 rounds), MAPO at 1.00% uplink reaches 92.73 ± 0.04 vs GaLore-FL's 91.67 ± 0.02 at 1.30% (+1.07, paired p=0.001p=0.001, ahead on every seed; FedAvg 91.6 ± 0.1), and at 0.10%, which GaLore-FL cannot reach, MAPO still reaches 92.47 ± 0.11.


(b) ViT-B/16 and Llama (Q2). Table 7's ViT-B/16 column now reports three seeds (best accuracy within 100 rounds):

Per-round uplink0.08%0.15%0.31%0.61%1.22%2.44%FedAvg
100 rounds61.6 ± 2.374.8 ± 1.977.1 ± 0.780.5 ± 0.381.7 ± 1.980.0 ± 1.286.2 ± 0.3
300 rounds84.7 ± 0.685.4 ± 1.6

With 300 rounds at 1.22%, MAPO reaches 85.4 ± 1.6 vs FedAvg's 86.2 ± 0.3 while using 3.7% of FedAvg's cumulative uplink. Llama now has 95% Wilson intervals on GSM8K:

kk2222^{22}2232^{23}2242^{24}2252^{25}2262^{26}2272^{27}FedAvg
GSM8K19.6 [17.6, 21.9]22.7 [20.5, 25.0]27.9 [25.5, 30.4]29.2 [26.8, 31.7]30.2 [27.8, 32.7]31.7 [29.2, 34.3]32.6 [30.1, 35.2]

At k=227k=2^{27}, the 0.9-point gap to FedAvg is within test-set error. ViT-L and Llama are single training runs, as the caption states.


(c) The 128-problem GSM8K subset (W2). The 128 problems were used only to monitor training; all reported Llama results are on the full 1,319-problem test set.


(d) GLUE (W2). Table 6 entries are means over three runs; the table now adds 95% Wilson half-widths: SST-2 1.2–2.4, QNLI 0.7–1.1, RTE 3.6–5.8, MNLI-m 0.6–0.9 and MNLI-mm 0.6–1.0 points.


(e) Extreme label skew (W2). Table 18 now reports mean ± std over three seeds with Welch tests. MAPO has the highest mean among the compressed methods at all 12 Dirichlet settings, significantly above both EvoFed and FedLoRU in 11, at 0.7–2.3% of FedAvg's uplink.


(f) Communication thresholds (W2). New Table 27 gives the cumulative uplink to 80/90/95% of FedAvg's accuracy, as % of FedAvg's, over three seeds at matched per-round uplink (MAPO k=420/416/41,412k=420/416/41{,}412):

MNIST 80 / 90 / 95%FMNIST 80 / 90 / 95%CIFAR-10 80 / 90 / 95%
MAPO4.7 / 6.0 / 8.95.3 / 8.5 / 10.23.0 / 2.8 / 2 of 3 seeds
Model-wide PowerSGD14.5 / 17.0 / 26.814.2 / 32.7 / not reached8.6 / not reached / not reached

In the revised paper, Tables 9 and 10 (Appendix D.1) give the GaLore-FL results, Table 7 and Appendix C.1 the ViT-B/16 seeds, Llama intervals and GSM8K clarification, Table 6 the GLUE intervals, Table 18 (Appendix F) the label skew, Table 27 (Appendix G.4) the thresholds, and Appendix E the statistical conventions.


We thank the reviewer again for the positive assessment and constructive feedback. The revision now documents the tuning protocol, adds three-seed results for the GaLore-FL and ViT-B/16 comparisons, provides paired tests and uncertainty estimates, and clarifies the GSM8K evaluation set.

We hope these additions address the reviewer's concerns. In light of the revised results and clarifications, we would be grateful if the reviewer would reconsider their score. We welcome any remaining questions.

thank you very much for detailing the configurations and hyperparameters! I appreciate it very much for updating Appendix E.1 and adding E.2 to provide the search grids/trials/checkpoint rule.

The added confidence interval with three seeds also helps a lot to justify the statistical performance of MAPO over other baselines!

I will keep my overall positive assesment!

We sincerely thank the reviewer for the follow-up and for the positive assessment. We are glad that the protocol details in Appendices E.1–E.2 and the three-seed confidence intervals are helpful, and we will keep them in the final version.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank the reviewers for detailed and constructive reviews. They describe MAPO as an "elegant, architecture-independent construction" that allows any integer budget k, "avoiding layer-shape rank floors", with "clear theoretical intuition" and "substantial empirical breadth" (EsfD), an "intuitive motivation and a clear, easy-to-follow methodological framework" (ZNf8), and "very concise", with "robust and comprehensive experiments" (vYix).

We revised the paper and added changes in blue. Each reply below gives the numbers directly; the PDF location is given only for checking.


New experiments and analyses

Requested byWhat was addedMain result
ZNf8Federated Aghajanyan et al.: {Fastfood, MAPO basis} × {fixed, refreshed} + SAID + dense, 3 CNN benchmarks, compression sweep, RoBERTa-base; exact per-round cost up to Llama-3.2-3B (App. G.2)On the CNNs a fixed intrinsic-dimension subspace is 2–3 points below MAPO at k/d ≈ 3.7% and 4.5–31.4 points below at k/d = 1–0.1%; on pretrained RoBERTa-base all arms are comparable. Refreshing the subspace is what matters, and MAPO makes it cheap: below 0.1% of local training compute at every scale
ZNf8Uplink to 80/90/95% of FedAvg for every Table-2 method (Table 4)MAPO is the cheapest compressed method at every target on every benchmark except Sent140 at 80%
vYixLayer-wise PowerSGD and Tucker-2 (± error feedback), two layer-wise MAPO variants (App. D.2)Layer-wise MAPO is comparable to model-wide MAPO; matrix/tensor decompositions cannot run at Table 2's MNIST/FMNIST budget (floors 7.1–16.6× MAPO's k = 128)
vYixLocal epochs E = 1–10; participation, availability and late joiners (App. G.3, F.1)MAPO has the highest mean final CIFAR-10 accuracy at every E and in every participation setting, at 3.6% of FedAvg's uplink
EsfD3 seeds / intervals for GaLore-FL, ViT-B, label skew, GLUE, GSM8K; tuning tables (App. C, C.1, D.1, E, F)With a decaying step size motivated by Cor. I.8(iii), MAPO is ahead of GaLore-FL by 2.8 / 7.3 / 9.3 points on CIFAR-100 at the 10% / 2% / 1% caps (p ≤ 0.003)

All new comparisons use one protocol for all methods they compare and report mean ± sample std over 3 seeds unless marked single run. The controlled matched-uplink comparisons use MAPO k = 420 / 416 / 41,412 on MNIST / FMNIST / CIFAR-10 (≈3.6–3.7% of the model), with every baseline set to that uplink.


For completeness, the remaining edits are below; none change the main results, methodology, theoretical analysis, or experimental setup.

  1. Abstract: "exceeding all evaluated baselines" → "exceeding the baselines", and the reconstruction vector is "regenerated every round from a seed shared by all clients".
  2. Sec. 2 credits intrinsic-dimension training, seed-generated factors, and redrawn subspaces; new App. K discusses them. Sec. 6 summarizes the new appendix studies in one sentence. We shortened a few redundant sentences to stay within the page limit.
  3. Reporting fixes: App. C says four GLUE tasks (five evaluations, counting MNLI-m/mm); GSM8K is the full 1,319-problem test set; Table 7 per-round Com. cells of ViT-L/Llama recomputed (e.g., 4.16 → 4.18%); Table 8 MAPO-40k row (15.6 MB/round); App. G uses 2√d = 37,799 for RoBERTa-large (about 5.8% above MAPO at k = 35,720); App. G.1's seed-variance sentence now matches Table 20.
  4. New appendix material: statistical-reporting conventions (App. E), per-dataset budgets (Table 16), search spaces and run counts (App. E.2, Table 17), a FedKSeed-style baseline (App. G.5), and a remark after Prop. I.10 showing that the frozen-basis floor holds for any frozen linear projection.

Dear Reviewers,

Thank you for your thoughtful reviews and contributions. The authors have now submitted their rebuttals. Please take some time to review their responses and assess whether your concerns have been adequately addressed. If appropriate, please update your comments and ratings accordingly.

Thank you for your time and continued efforts.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.