- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SeFoRA (Sketch-Aggregated Federated LoRA), a novel algorithm that simultaneously addresses two core challenges in Federated Learning (FL) with LoRA: Rank Heterogeneity and Bilinear Mismatch. Unlike prior methods that rely on zero-padding (which incurs mathematical errors) or dense reconstruction in the full parameter space (d×s) at the server (which incurs massive computational costs), SEFORA adopts the linear matrix sketching technique by Tropp et al. (2017). Clients compress their updates into small, fixed-size sketches (Y,Z). The federator aggregates these in the sketch space and reconstructs a low-rank global adapter, completely bypassing bilinear errors. The authors theoretically prove an O(T^(-1/2) ) convergence rate for non-convex loss functions and demonstrate competitive or superior performance against existing SOTA baselines in various experiments using language and vision foundation models.
Critical strengths
-
Simple, rank-agnostic aggregation for federated learning: Fixed-size linear sketches allow updates from clients with arbitrary ranks to be aggregated by simple summation, without zero-padding or replication heuristics. In principle, this structure is compatible with secure aggregation.
-
Solid theoretical guarantees: The theoretical analysis is academically rigorous. It explicitly separates three major error sources (sketching approximation error, rank truncation information loss, and client data heterogeneity) into the convergence bound, providing a strict O(T^(-1/2) ) convergence proof.
Critical weaknesses
The description of flexible upload is inconsistent, and it is unclear whether it was used in the experiments.
-
Algorithm 1 (line 14) specifies that every client always uploads sketches. Section 3.1 states that only clients with r_{I,l} ≪ r_l may upload their adapters instead, while Section 5.1 states that each client uploads whichever of the two is smaller. However, by the inequality in W1.1, a literal application of the Section 5.1 rule means every client would upload adapters, and client-side sketching would never be used. The paper does not specify which rule is intended, nor what each client actually uploaded in each experiment.
-
There is no ablation isolating the source of the performance gains. SeFoRA combines three components: change-relative aggregation, a single-pass randomized SVD of the aggregate, and balanced re-factorization. Of these, the component that prevents dilution of directions unseen by low-rank clients is the change-relative aggregation, whereas sketching merely approximates an exact truncation. This interpretation is consistent with Table 3, where FlexLoRA, which performs an exact truncation, achieves the highest accuracy on all three datasets. Because the paper does not compare against a variant such as change-relative aggregation combined with an exact factored SVD, it remains unclear which component drives the performance.
Questions for authors
-
The description of flexible upload is inconsistent. Algorithm 1 always uploads sketches, Section 3.1 states that only clients with upload adapters, and Section 5.1 states that each client uploads “whichever is smaller.” However, since , the inequality holds for every client. A literal application of the Section 5.1 rule therefore means every client uploads adapters. In that case, client-side sketching is never used and the server receives every individual adapter, which also conflicts with the claimed compatibility with secure aggregation. Could you clarify which rule is intended and which scheme was actually used in each experiment?
-
SeFoRA combines three components: change-relative aggregation, a single-pass randomized SVD of the aggregate, and balanced re-factorization. Of these, the component that prevents the dilution of directions unseen by low-rank clients is change-relative aggregation, whereas sketching only approximates an exact rank- truncation. This is consistent with Table 3, where FlexLoRA, which performs an exact truncation, outperforms SeFoRA on all three datasets. However, since the paper does not compare against a variant that combines change-relative aggregation with an exact factored truncated SVD without sketching, nor against a variant that sketches the final local updates instead of the changes, it remains unclear which component drives the performance. Could you report these ablations and clarify whether the gap to FlexLoRA in Table 3 is due to sketching error?
5 comments
W2/Q2: Ablation to clarify the gap to FlexLoRA
Response. We have added the ablation (Table 1), and it shows that the gap comes from the change-relative aggregation, not the sketching error. We therefore revised the algorithm by removing the error-compensation step, i.e., the clients now sketch their final local adapters directly, rather than the relative change . This results in a cleaner, simpler algorithm. The federator no longer needs to sketch the global adapter, and clients sketch once per round rather than twice, reducing client sketching time. We revised the algorithm and the theory, and reran all main experiments. The revised experiments (Table 2-4) show that SeFoRA outperforms the other methods, especially in the extreme cases of rank and data heterogeneity.
Table 1. Accuracy (%) on CIFAR-100 under standard () and extreme () rank heterogeneity.
| Variant | Aggregation target | Reconstruction | Factors | Standard | Extreme |
|---|---|---|---|---|---|
| FlexLoRA | direct | exact SVD | asymmetric | 91.1 ± 0.1 | 91.0 ± 0.3 |
| V1 | direct | exact SVD | balanced | 91.3 ± 0.1 | 91.4 ± 0.1 |
| V2 | change-relative | exact SVD | asymmetric | 90.1 ± 0.2 | 89.5 ± 0.5 |
| V3 | change-relative | exact SVD | balanced | 90.1 ± 0.2 | 89.5 ± 0.6 |
| SeFoRA (original) | change-relative | sketch (randomized SVD) | balanced | 91.0 ± 0.1 | 89.4 ± 0.7 |
| SeFoRA (revised) | direct | sketch (randomized SVD) | balanced | 91.4 ± 0.1 | 91.4 ± 0.2 |
Table 2. Accuracy (%) on DomainNet using ViT, with .
| α | Method | Clipart | Infograph | Painting | Quickdraw | Real | Sketch | Average |
|---|---|---|---|---|---|---|---|---|
| 0.1 | FlexLoRA | 63.0 ± 0.5 | 31.3 ± 0.4 | 61.1 ± 0.3 | 32.6 ± 0.2 | 78.2 ± 0.2 | 53.9 ± 0.5 | 53.4 ± 0.3 |
| 0.1 | Fed-PLoRA | 68.1 ± 0.7 | 33.6 ± 0.3 | 63.5 ± 0.4 | 34.7 ± 0.2 | 78.9 ± 0.3 | 57.7 ± 0.1 | 56.1 ± 0.1 |
| 0.1 | FSLoRA | 66.7 ± 0.2 | 32.9 ± 0.4 | 63.4 ± 0.5 | 38.3 ± 0.4 | 80.4 ± 0.3 | 56.9 ± 0.1 | 56.4 ± 0.3 |
| 0.1 | FedHL | 52.1 ± 1.0 | 25.3 ± 0.4 | 49.6 ± 1.0 | 21.4 ± 1.0 | 59.9 ± 0.3 | 44.3 ± 0.9 | 42.1 ± 0.7 |
| 0.1 | SeFoRA | 67.6 ± 0.2 | 33.6 ± 0.2 | 63.6 ± 0.3 | 40.4 ± 0.4 | 80.6 ± 0.2 | 57.8 ± 0.2 | 57.3 ± 0.1 |
| 0.5 | FlexLoRA | 63.4 ± 0.2 | 32.0 ± 0.3 | 61.7 ± 0.4 | 33.5 ± 0.1 | 78.6 ± 0.3 | 54.3 ± 0.3 | 53.9 ± 0.1 |
| 0.5 | Fed-PLoRA | 69.2 ± 0.2 | 34.4 ± 0.4 | 64.3 ± 0.3 | 36.0 ± 0.3 | 79.7 ± 0.3 | 58.6 ± 0.1 | 57.1 ± 0.1 |
| 0.5 | FSLoRA | 67.5 ± 0.3 | 33.4 ± 0.3 | 64.1 ± 0.2 | 39.2 ± 0.4 | 80.8 ± 0.2 | 57.5 ± 0.4 | 57.1 ± 0.2 |
| 0.5 | FedHL | 64.0 ± 0.3 | 31.1 ± 0.6 | 59.2 ± 1.3 | 27.5 ± 0.6 | 72.0 ± 1.6 | 53.3 ± 1.4 | 51.2 ± 0.9 |
| 0.5 | SeFoRA | 67.8 ± 0.1 | 33.8 ± 0.1 | 64.4 ± 0.1 | 41.4 ± 0.2 | 80.9 ± 0.2 | 58.1 ± 0.1 | 57.7 ± 0.1 |
Table 3. Accuracy (%) on GSM8K and the GLUE tasks, all with .
| Method | GSM8K (Llama) | GSM8K (Qwen) | SST-2 | QNLI | QQP | MNLI |
|---|---|---|---|---|---|---|
| FlexLoRA | 34.5 ± 0.8 | 56.0 ± 0.4 | 95.3 ± 0.3 | 90.4 ± 1.3 | 70.6 ± 12.4 | 85.1 ± 2.5 |
| Fed-PLoRA | 30.1 ± 0.8 | 55.3 ± 0.5 | 94.1 ± 0.9 | 88.4 ± 1.3 | 69.5 ± 11.0 | 86.3 ± 1.0 |
| FSLoRA | 36.3 ± 1.1 | 55.7 ± 0.8 | 95.0 ± 0.1 | 90.9 ± 1.1 | 70.9 ± 13.5 | 87.0 ± 1.0 |
| FedHL | 21.5 ± 4.6 | 55.5 ± 0.7 | 93.8 ± 1.6 | 82.2 ± 5.9 | 75.4 ± 7.7 | 77.0 ± 10.6 |
| SeFoRA | 36.4 ± 0.4 | 56.3 ± 0.4 | 95.2 ± 0.1 | 92.0 ± 1.5 | 71.9 ± 11.2 | 86.4 ± 2.0 |
Table 4. Accuracy (%) on CIFAR-100, Tiny ImageNet, and Food-101 under extreme and standard rank heterogeneity.
| Rank heterogeneity | Method | CIFAR-100 | Tiny ImageNet | Food-101 |
|---|---|---|---|---|
| Extreme | FlexLoRA | 91.0 ± 0.3 | 88.0 ± 0.2 | 84.3 ± 0.5 |
| Extreme | Fed-PLoRA | 87.1 ± 0.3 | 84.9 ± 0.7 | 83.2 ± 1.1 |
| Extreme | FSLoRA | 88.9 ± 0.3 | 86.3 ± 0.2 | 83.3 ± 0.9 |
| Extreme | FedHL | 85.0 ± 0.6 | 82.2 ± 0.3 | 76.8 ± 3.2 |
| Extreme | SeFoRA | 91.4 ± 0.2 | 88.7 ± 0.1 | 86.3 ± 0.4 |
| Standard | FlexLoRA | 91.1 ± 0.2 | 87.9 ± 0.1 | 84.2 ± 0.6 |
| Standard | Fed-PLoRA | 90.8 ± 0.2 | 88.2 ± 0.2 | 86.6 ± 0.2 |
| Standard | FSLoRA | 91.0 ± 0.1 | 88.4 ± 0.2 | 86.2 ± 0.8 |
| Standard | FedHL | 83.4 ± 1.2 | 79.5 ± 1.3 | 75.6 ± 2.9 |
| Standard | SeFoRA | 91.4 ± 0.1 | 88.9 ± 0.1 | 86.6 ± 0.5 |
I appreciate the authors' response.
It seems that the new manuscript doesn't reflect the response.
My remaining concern from the manuscript is still the limited experiments, as the main paper contain only 3 tables and not that many experiments exist in the appendix. Also, given the marginal gain against FSLoRA, I'm not clear whether the paper meets the acceptance bar of ICLR.
We thank Reviewer 683P for the request for ablation studies. It exposed a design choice that hurt accuracy.
Response. We thank the Reviewer for pointing it out. The sentence in Section 5.1 was not precise. To avoid confusion, we clarify the intended rule as follows. By default, every client sketches its own update and uploads the sketches (Algorithm 1). The case ri,l≪rl was intended for the clients most affected by the fixed sketch size. The option of uploading factors offers particularly large relative communication savings for those low-rank clients. When ri,l is close to rl, sketch upload incurs only a modest relative overhead. Since factors are always smaller than sketches, if minimizing uplink communication is the sole criterion, all clients should upload their factors. However, factor upload transfers the computation to the federator. Therefore, the choice reflects a trade-off between communication cost and federator computation. In our simulations, uploading factors for server-side sketching and uploading client-computed sketches are mathematically equivalent. With shared sketching matrices Ω,Ψ, both choices produce the same aggregated sketches and reconstructed global adapters. Therefore, the reported accuracy results apply to both upload options.
Regarding secure aggregation, since privacy is not the main concern in this work, we assume that the federator may access individual clients' factors. When this assumption is acceptable, clients can shift sketch computation to the federator. However, if privacy-preserving aggregation is required, either the clients must compute the sketches, so that secure aggregation is directly compatible, increasing communication costs, or additional protection is necessary.