acceptodds
Under review as a conference paper at ICLR 2027

SeFoRA: Sketch-Aggregated Federated LoRA with Heterogeneous Client Ranks

Abstract

We consider federated parameter-efficient fine-tuning of foundation models, e.g., LLMs and LVMs, using low-rank adaptation (LoRA;  Hu et al. 2022). Combining LoRA with federated learning introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose , a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator. As a result, overcomes the rank heterogeneity challenge and alleviates the bilinear mismatch. We establish convergence to a neighborhood of a first-order stationary point at rate . Extensive numerical experiments on language and vision models showcase the performance of our algorithm compared to existing methods.

Primary area:
optimization
Keywords:
parameter efficient fine-tuning, low-rank adaptation, federated learning, client rank heterogeneity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_683POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SeFoRA (Sketch-Aggregated Federated LoRA), a novel algorithm that simultaneously addresses two core challenges in Federated Learning (FL) with LoRA: Rank Heterogeneity and Bilinear Mismatch. Unlike prior methods that rely on zero-padding (which incurs mathematical errors) or dense reconstruction in the full parameter space (d×s) at the server (which incurs massive computational costs), SEFORA adopts the linear matrix sketching technique by Tropp et al. (2017). Clients compress their updates into small, fixed-size sketches (Y,Z). The federator aggregates these in the sketch space and reconstructs a low-rank global adapter, completely bypassing bilinear errors. The authors theoretically prove an O(T^(-1/2) ) convergence rate for non-convex loss functions and demonstrate competitive or superior performance against existing SOTA baselines in various experiments using language and vision foundation models.

Critical strengths
  1. Simple, rank-agnostic aggregation for federated learning: Fixed-size linear sketches allow updates from clients with arbitrary ranks to be aggregated by simple summation, without zero-padding or replication heuristics. In principle, this structure is compatible with secure aggregation.

  2. Solid theoretical guarantees: The theoretical analysis is academically rigorous. It explicitly separates three major error sources (sketching approximation error, rank truncation information loss, and client data heterogeneity) into the convergence bound, providing a strict O(T^(-1/2) ) convergence proof.

Critical weaknesses

The description of flexible upload is inconsistent, and it is unclear whether it was used in the experiments.

  1. Algorithm 1 (line 14) specifies that every client always uploads sketches. Section 3.1 states that only clients with r_{I,l} ≪ r_l may upload their adapters instead, while Section 5.1 states that each client uploads whichever of the two is smaller. However, by the inequality in W1.1, a literal application of the Section 5.1 rule means every client would upload adapters, and client-side sketching would never be used. The paper does not specify which rule is intended, nor what each client actually uploaded in each experiment.

  2. There is no ablation isolating the source of the performance gains. SeFoRA combines three components: change-relative aggregation, a single-pass randomized SVD of the aggregate, and balanced re-factorization. Of these, the component that prevents dilution of directions unseen by low-rank clients is the change-relative aggregation, whereas sketching merely approximates an exact truncation. This interpretation is consistent with Table 3, where FlexLoRA, which performs an exact truncation, achieves the highest accuracy on all three datasets. Because the paper does not compare against a variant such as change-relative aggregation combined with an exact factored SVD, it remains unclear which component drives the performance.

Questions for authors
  1. The description of flexible upload is inconsistent. Algorithm 1 always uploads sketches, Section 3.1 states that only clients with ri,l≪rlr_{i,l} \ll r_l upload adapters, and Section 5.1 states that each client uploads “whichever is smaller.” However, since ri,l≤rl<klr_{i,l} \le r_l < k_l, the inequality (d+s)ri,l<drl+kls(d+s)r_{i,l} < dr_l + k_l s holds for every client. A literal application of the Section 5.1 rule therefore means every client uploads adapters. In that case, client-side sketching is never used and the server receives every individual adapter, which also conflicts with the claimed compatibility with secure aggregation. Could you clarify which rule is intended and which scheme was actually used in each experiment?

  2. SeFoRA combines three components: change-relative aggregation, a single-pass randomized SVD of the aggregate, and balanced re-factorization. Of these, the component that prevents the dilution of directions unseen by low-rank clients is change-relative aggregation, whereas sketching only approximates an exact rank-rr truncation. This is consistent with Table 3, where FlexLoRA, which performs an exact truncation, outperforms SeFoRA on all three datasets. However, since the paper does not compare against a variant that combines change-relative aggregation with an exact factored truncated SVD without sketching, nor against a variant that sketches the final local updates Bi(t,E)Ai(t,E)B_i^{(t,E)}A_i^{(t,E)} instead of the changes, it remains unclear which component drives the performance. Could you report these ablations and clarify whether the gap to FlexLoRA in Table 3 is due to sketching error?

5 comments

We thank Reviewer 683P for the request for ablation studies. It exposed a design choice that hurt accuracy.

W1/Q1: The description of flexible upload is inconsistent. Algorithm 1 always uploads sketches, Section 3.1 states that only clients with ri,l≪rlr_{i,l} \ll r_l upload adapters, and Section 5.1 states that each client uploads “whichever is smaller.” However, since ri,l≤rl<klr_{i,l}\leq r_l<k_l, the inequality (d+s)ri,l<drl+kls(d+s)r_{i,l}<dr_l+k_l s holds for every client. A literal application of the Section 5.1 rule therefore means every client uploads adapters. In that case, client-side sketching is never used and the server receives every individual adapter, which also conflicts with the claimed compatibility with secure aggregation. Could you clarify which rule is intended and which scheme was actually used in each experiment?

Response. We thank the Reviewer for pointing it out. The sentence in Section 5.1 was not precise. To avoid confusion, we clarify the intended rule as follows. By default, every client sketches its own update and uploads the sketches (Algorithm 1). The case ri,l≪rlr_{i,l}\ll r_l was intended for the clients most affected by the fixed sketch size. The option of uploading factors offers particularly large relative communication savings for those low-rank clients. When ri,lr_{i,l} is close to rlr_l, sketch upload incurs only a modest relative overhead. Since factors are always smaller than sketches, if minimizing uplink communication is the sole criterion, all clients should upload their factors. However, factor upload transfers the computation to the federator. Therefore, the choice reflects a trade-off between communication cost and federator computation. In our simulations, uploading factors for server-side sketching and uploading client-computed sketches are mathematically equivalent. With shared sketching matrices Ω,Ψ\Omega,\Psi, both choices produce the same aggregated sketches and reconstructed global adapters. Therefore, the reported accuracy results apply to both upload options.

Regarding secure aggregation, since privacy is not the main concern in this work, we assume that the federator may access individual clients' factors. When this assumption is acceptable, clients can shift sketch computation to the federator. However, if privacy-preserving aggregation is required, either the clients must compute the sketches, so that secure aggregation is directly compatible, increasing communication costs, or additional protection is necessary.

W2/Q2: Ablation to clarify the gap to FlexLoRA

Response. We have added the ablation (Table 1), and it shows that the gap comes from the change-relative aggregation, not the sketching error. We therefore revised the algorithm by removing the error-compensation step, i.e., the clients now sketch their final local adapters Bi(t,E)Ai(t,E)\mathbf{B}_i^{(t,E)}\mathbf{A}_i^{(t,E)} directly, rather than the relative change Bi(t,E)Ai(t,E)−Bi(t,0)Ai(t,0)\mathbf{B}_i^{(t,E)}\mathbf{A}_i^{(t,E)}-\mathbf{B}_i^{(t,0)}\mathbf{A}_i^{(t,0)}. This results in a cleaner, simpler algorithm. The federator no longer needs to sketch the global adapter, and clients sketch once per round rather than twice, reducing client sketching time. We revised the algorithm and the theory, and reran all main experiments. The revised experiments (Table 2-4) show that SeFoRA outperforms the other methods, especially in the extreme cases of rank and data heterogeneity.

Table 1. Accuracy (%) on CIFAR-100 under standard (rmax⁡=16r_{\max}=16) and extreme (rmax⁡=256r_{\max}=256) rank heterogeneity.

VariantAggregation targetReconstructionFactorsStandardExtreme
FlexLoRAdirectexact SVDasymmetric91.1 ± 0.191.0 ± 0.3
V1directexact SVDbalanced91.3 ± 0.191.4 ± 0.1
V2change-relativeexact SVDasymmetric90.1 ± 0.289.5 ± 0.5
V3change-relativeexact SVDbalanced90.1 ± 0.289.5 ± 0.6
SeFoRA (original)change-relativesketch (randomized SVD)balanced91.0 ± 0.189.4 ± 0.7
SeFoRA (revised)directsketch (randomized SVD)balanced91.4 ± 0.191.4 ± 0.2

Table 2. Accuracy (%) on DomainNet using ViT, with rmax⁡=16r_{\max}=16.

αMethodClipartInfographPaintingQuickdrawRealSketchAverage
0.1FlexLoRA63.0 ± 0.531.3 ± 0.461.1 ± 0.332.6 ± 0.278.2 ± 0.253.9 ± 0.553.4 ± 0.3
0.1Fed-PLoRA68.1 ± 0.733.6 ± 0.363.5 ± 0.434.7 ± 0.278.9 ± 0.357.7 ± 0.156.1 ± 0.1
0.1FSLoRA66.7 ± 0.232.9 ± 0.463.4 ± 0.538.3 ± 0.480.4 ± 0.356.9 ± 0.156.4 ± 0.3
0.1FedHL52.1 ± 1.025.3 ± 0.449.6 ± 1.021.4 ± 1.059.9 ± 0.344.3 ± 0.942.1 ± 0.7
0.1SeFoRA67.6 ± 0.233.6 ± 0.263.6 ± 0.340.4 ± 0.480.6 ± 0.257.8 ± 0.257.3 ± 0.1
0.5FlexLoRA63.4 ± 0.232.0 ± 0.361.7 ± 0.433.5 ± 0.178.6 ± 0.354.3 ± 0.353.9 ± 0.1
0.5Fed-PLoRA69.2 ± 0.234.4 ± 0.464.3 ± 0.336.0 ± 0.379.7 ± 0.358.6 ± 0.157.1 ± 0.1
0.5FSLoRA67.5 ± 0.333.4 ± 0.364.1 ± 0.239.2 ± 0.480.8 ± 0.257.5 ± 0.457.1 ± 0.2
0.5FedHL64.0 ± 0.331.1 ± 0.659.2 ± 1.327.5 ± 0.672.0 ± 1.653.3 ± 1.451.2 ± 0.9
0.5SeFoRA67.8 ± 0.133.8 ± 0.164.4 ± 0.141.4 ± 0.280.9 ± 0.258.1 ± 0.157.7 ± 0.1

Table 3. Accuracy (%) on GSM8K and the GLUE tasks, all with rmax⁡=16r_{\max}=16.

MethodGSM8K (Llama)GSM8K (Qwen)SST-2QNLIQQPMNLI
FlexLoRA34.5 ± 0.856.0 ± 0.495.3 ± 0.390.4 ± 1.370.6 ± 12.485.1 ± 2.5
Fed-PLoRA30.1 ± 0.855.3 ± 0.594.1 ± 0.988.4 ± 1.369.5 ± 11.086.3 ± 1.0
FSLoRA36.3 ± 1.155.7 ± 0.895.0 ± 0.190.9 ± 1.170.9 ± 13.587.0 ± 1.0
FedHL21.5 ± 4.655.5 ± 0.793.8 ± 1.682.2 ± 5.975.4 ± 7.777.0 ± 10.6
SeFoRA36.4 ± 0.456.3 ± 0.495.2 ± 0.192.0 ± 1.571.9 ± 11.286.4 ± 2.0

Table 4. Accuracy (%) on CIFAR-100, Tiny ImageNet, and Food-101 under extreme and standard rank heterogeneity.

Rank heterogeneityMethodCIFAR-100Tiny ImageNetFood-101
ExtremeFlexLoRA91.0 ± 0.388.0 ± 0.284.3 ± 0.5
ExtremeFed-PLoRA87.1 ± 0.384.9 ± 0.783.2 ± 1.1
ExtremeFSLoRA88.9 ± 0.386.3 ± 0.283.3 ± 0.9
ExtremeFedHL85.0 ± 0.682.2 ± 0.376.8 ± 3.2
ExtremeSeFoRA91.4 ± 0.288.7 ± 0.186.3 ± 0.4
StandardFlexLoRA91.1 ± 0.287.9 ± 0.184.2 ± 0.6
StandardFed-PLoRA90.8 ± 0.288.2 ± 0.286.6 ± 0.2
StandardFSLoRA91.0 ± 0.188.4 ± 0.286.2 ± 0.8
StandardFedHL83.4 ± 1.279.5 ± 1.375.6 ± 2.9
StandardSeFoRA91.4 ± 0.188.9 ± 0.186.6 ± 0.5

I appreciate the authors' response.

It seems that the new manuscript doesn't reflect the response.

My remaining concern from the manuscript is still the limited experiments, as the main paper contain only 3 tables and not that many experiments exist in the appendix. Also, given the marginal gain against FSLoRA, I'm not clear whether the paper meets the acceptance bar of ICLR.

Reviewer_QKTiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies federated LoRA with heterogeneous client and layer ranks. SeFoRA has each client upload two linear sketches of its change in the full LoRA update. Because sketching is linear, their weighted sum is a sketch of the aggregated update, avoiding both dimension mismatch and the cross terms caused by averaging LoRA factors. The server reconstructs a rank-constrained update and sends each client a rank-truncated version. The paper provides a convergence bound with separate sketching, rank-truncation, and data-heterogeneity terms, and evaluates the method on GLUE, GSM8K, and several vision datasets.

Critical strengths

The experiments span language, reasoning, and vision tasks and include domain/label heterogeneity, extreme rank profiles, partial participation, client-count scaling, rank and sketch-size ablations, and server timing.

Critical weaknesses
  1. For non-zero spectral-tail error, Corollary 1 gives an O(T−1/2)O(T^{−1/2}) transient plus a fixed stationarity neighborhood, not convergence to zero. The paper neither states that limiting neighborhood compactly nor estimates its sketch-tail, truncation, heterogeneity, or qmin terms. Assumption 5 is unverified, and the analysis does not cover the Adam optimizer used in every experiment.

  2. A sketch upload costs dr+ksdr+ks entries per layer and is determined by the maximum rank, so low-rank clients may communicate substantially more than their LoRA factors. In the extreme ViT setting (d=s=768, r=256, k=288), this is about entries, close to a dense update. The proposed factor-versus-sketch upload rule shifts work to the server and changes the secure-aggregation story, but its use and cost are not reported. Table 7 covers only one setting and shows end-to-end time roughly tied with FSLoRA. No per-rank bytes, memory, client sketching time, or accuracy-cost Pareto comparison is provided.

  3. SeFoRA is competitive, but FlexLoRA is best on all three extreme-rank vision tasks, SeFoRA wins none of the four GLUE columns, and some gains over FSLoRA are comparable to three-seed variability. The paper also does not explain how the trainable classification head is synchronized or how weights are normalized under partial participation.

Questions for authors
  1. Can you state Corollary 1 explicitly as a transient plus its limiting neighborhood, estimate the relevant terms or reconstruction error in the experiments, and either justify Adam theoretically or provide matched SGD results?

  2. Please report uplink/downlink bytes by method and rank tier, total communication, client sketching time, peak memory, and server cost. Which runs use the flexible upload rule, and can that rule retain secure aggregation without exposing individual factors?

  3. How is the classification head initialized, aggregated, communicated, and evaluated? How is the learning rate selected, with what validation and tuning budget? Under partial participation, are selected-client weights renormalized or importance-weighted?

5 comments

We thank Reviewer QKTi for carefully reading the theoretical derivations and for the insightful questions.

W1: For non-zero spectral-tail error, Corollary 1 gives an O(T−1/2)\mathcal{O}(T^{-1/2}) transient plus a fixed stationarity neighborhood, not convergence to zero. The paper neither states that limiting neighborhood compactly nor estimates its sketch-tail, truncation, heterogeneity, or qmin terms. Assumption 5 is unverified, and the analysis does not cover the Adam optimizer used in every experiment.

Q1: Can you state Corollary 1 explicitly as a transient plus its limiting neighborhood, estimate the relevant terms or reconstruction error in the experiments, and either justify Adam theoretically or provide matched SGD results?

We note that the comment from Reviewer 683P led to a change in the method: the clients now sketch their final local adapters Bi(t,E)Ai(t,E)\mathbf{B}_i^{(t,E)}\mathbf{A}_i^{(t,E)} directly, rather than the relative change Bi(t,E)Ai(t,E)−Bi(t,0)Ai(t,0)\mathbf{B}_i^{(t,E)}\mathbf{A}_i^{(t,E)}-\mathbf{B}_i^{(t,0)}\mathbf{A}_i^{(t,0)}. We revised SeFoRA by removing the compensation step and revised the convergence analysis accordingly. Specifically, the federator's new global update yields

ΔWSeFoRA(t+1)−ΔW(t)=∑i∈[N]piΔWi,ri(t,E)−∑i∈[N]piΔWi,ri(t,0)+Esketch(t+1)−Etrunc(t),\begin{aligned}\Delta \mathbf{\mathtt{W}}_{\text{SeFoRA}}^{(t+1)} - \Delta \mathbf{\mathtt{W}}^{(t)} &= \sum_{i \in [N]} p_i \Delta \mathbf{\mathtt{W}}_{i,r_i}^{(t,E)} - \sum_{i \in [N]} p_i \Delta \mathbf{\mathtt{W}}_{i,r_i}^{(t,0)} + \mathcal{E}_{\text{sketch}}^{(t+1)} - \mathcal{E}_{\text{trunc}}^{(t)},\end{aligned}

and Etrunc(t)=ΔW(t)−∑i∈[N]piΔWi,ri(t,0)\mathcal{E}_{\text{trunc}}^{(t)}=\Delta \mathbf{\mathtt{W}}^{(t)}-\sum_{i \in [N]} p_i \Delta \mathbf{\mathtt{W}}_{i,r_i}^{(t,0)}. Thus, Theorem 1 and Corollary 1 keep their form, and the floor S1S_1 now contains both Esketch\mathcal{E}_{\text{sketch}} and Etrunc\mathcal{E}_{\text{trunc}}, i.e.,

S_1^{(t+1)} &\triangleq \chi \cdot \mathbb{E}_t \Big\\| \mathcal{E}_{\text{sketch}}^{(t+1)} - \mathcal{E}_{\text{trunc}}^{(t)} \Big\\|_F + 2\mu \mathbb{E}_t \Big\\| \mathcal{E}_{\text{sketch}}^{(t+1)} - \mathcal{E}_{\text{trunc}}^{(t)} \Big\\|_F^2. \end{aligned}$$ For the refined algorithm, the convergence bound consists of an $\mathcal{O}(T^{-1/2})$ transient term and a $T$-independent residual term that depends on sketching error, clients' rank truncation, and client heterogeneity. Regarding Assumption 5, we measured the absolute spectral tail energy of the exact aggregate on CIFAR-100 over 300 rounds. Table 1 reports $\sum_l \tau^2_{\varrho_l+1}(\mathbf{V}_l^{(t+1)})$, summed over the 24 adapted matrices. Under Adam, the early- and late-round means are similar, consistent with Assumption 5. Under SGD, the tails are several orders of magnitude smaller. **Table 1.** Tail energy $\sum_l\tau^2_{\varrho_l+1}(\mathbf{V}_l^{(t+1)})$ of the exact aggregate on CIFAR-100, for the fixed $\varrho_l$ in Assumption~5. | Setting | $\varrho_l$ | Mean tail energy, rounds 1–100 | Mean tail energy, rounds 201–300 | |---|---|---|---| | Adam, standard | $r_l-2$ | 0.107 | 0.099 | | Adam, extreme | $\lfloor r_l/2\rfloor$ | 0.119 | 0.116 | | SGD, standard | $r_l-2$ | $5.2\times10^{-6}$ | $1.7\times10^{-5}$ | | SGD, extreme | $\lfloor r_l/2\rfloor$ | $1.5\times10^{-9}$ | $3.5\times10^{-9}$ | Like prior work, our theorem analyzes SGD. The main experiments use Adam due to its empirical advantages in optimization. To address this gap, we also conducted matched SGD experiments on CIFAR-100, TinyImageNet, and Food101 under both extreme and standard rank-heterogeneity settings. The results (Table 2) show that SGD converges more slowly and continues improving at round 300, yet it reaches nearly the same accuracy. **Table 2.** SeFoRA with Adam and SGD with 300 rounds. | Dataset | Standard: Adam | Standard: SGD | Extreme: Adam | Extreme: SGD | |---|---:|---:|---:|---:| | CIFAR-100 | 91.39 ± 0.09 | 90.64 ± 0.05 | 91.39 ± 0.15 | 90.70 ± 0.06 | | Tiny-ImageNet | 88.93 ± 0.04 | 88.50 ± 0.05 | 88.72 ± 0.11 | 88.55 ± 0.05 | | Food-101 | 86.60 ± 0.50 | 85.93 ± 0.08 | 86.34 ± 0.44 | 86.18 ± 0.07 |

W2/Q2:

Response. We agree that fixed-size sketches can increase communication for low-rank clients. We report the costs, memory usage, and time. The standard tiers contain 7, 15, and 8 clients, respectively; the extreme tiers contain 10 clients each with tiers 8,168, 16 and 256256. Note that the table only reports a uniform upload policy; a mixed policy can be calculated by summing the corresponding payload for each tier.

Table 3 shows that, with sketch upload, SeFoRA reduces the federator time per round and the federator peak memory compared with FlexLoRA, at the expense of increased communication. The total time that 30 clients spend per round on sketching is relatively light compared with the local training time. In addition, the revised SeFoRA reaches higher accuracy than FlexLoRA in both standard and extreme settings. So SeFoRA with sketch upload outperforms FlexLoRA in terms of accuracy, computation, and memory.

In terms of communication, Table 4 reports the uplink and downlink bytes for different tiers in the standard and extreme rank heterogeneity settings, and Table 5 reports the overall communication per round. Uploading sketches increases communication cost, especially when the clients' rank heterogeneity is extreme. Thus, in the extreme setting, clients with ri,l≪rlr_{i,l} \ll r_l may upload their factors to the federator. Factor upload reduces uplink communication by transferring sketch construction from clients to the federator. Since the sketch is linear and Ω\Omega and Ψ\Psi are shared, computing the sketch on the server or the client would yield identical accuracy.

Finally, as privacy is not the main concern in this work, we assume that the federator may access individual client factors. When this assumption is acceptable, clients can shift sketch computation to the federator. However, if privacy-preserving aggregation is required, either the clients must compute the sketches, so that secure aggregation is directly compatible, increasing communication costs, or additional protection is necessary.

Table 3. Per-round computation time and server memory measurements on CIFAR-100, with all clients uploading their sketches in SeFoRA. Server time includes aggregation and construction of the next global adapters. Compute time is the sum of server time, total client sketching time, and total client training time. The standard and extreme settings were run on different GPUs (RTX PRO 4500 and RTX PRO 5000), so compare methods only within each setting.

SettingMethodAccuracy (%)Server (s/round)Total client sketch (s/round)Total client train (s/round)Compute (s/round)Server peak (MiB)
StandardFlexLoRA91.1 ± 0.20.21206.486.69286.8
StandardSeFoRA91.4 ± 0.10.0200.0586.516.59240.1
ExtremeFlexLoRA91.0 ± 0.30.88905.116.00411.8
ExtremeSeFoRA91.4 ± 0.20.1860.2005.125.51270.8

Table 4. Uplink and downlink bytes (MiB) in both standard and extreme rank-heterogeneity settings. The tier maximum denotes the maximum layer rank.

Rank heterogeneityTier max rankFactor uplink/clientSketch uplink/clientDownload/client
Standard80.8441.8280.844
Standard121.2661.8281.266
Standard161.6881.8281.688
Extreme80.84428.7170.844
Extreme161.68828.7171.688
Extreme25627.00028.71727.000

Table 5. Accuracy and computed communication (MiB) per round on CIFAR-100 (all 30 clients, adapters and head). Standard: rmax⁡=16r_{\max}=16; extreme: rmax⁡=256r_{\max}=256.

MethodStandard accuracy (%)Standard up / down (MiB)Standard two-way (MiB)Extreme accuracy (%)Extreme up / down (MiB)Extreme two-way (MiB)
FlexLoRA91.1 ± 0.242.8 / 42.885.691.0 ± 0.3299.7 / 299.7599.4
Fed-PLoRA90.8 ± 0.242.8 / 55.097.887.1 ± 0.3299.7 / 814.41114.1
FSLoRA91.0 ± 0.142.8 / 55.097.888.9 ± 0.3299.7 / 814.41114.1
FedHL83.4 ± 1.242.8 / 42.885.685.0 ± 0.6299.7 / 299.7599.4
SeFoRA, factor upload91.4 ± 0.142.8 / 42.885.691.4 ± 0.2299.7 / 299.7599.4
SeFoRA, sketch upload91.4 ± 0.159.2 / 42.8102.091.4 ± 0.2865.9 / 299.71165.6

W3: SeFoRA is competitive, but FlexLoRA is best on all three extreme-rank vision tasks, SeFoRA wins none of the four GLUE columns, and some gains over FSLoRA are comparable to three-seed variability. The paper also does not explain how the trainable classification head is synchronized or how weights are normalized under partial participation.

Q3: How is the classification head initialized, aggregated, communicated, and evaluated? How is the learning rate selected, with what validation and tuning budget? Under partial participation, are selected-client weights renormalized or importance-weighted?

Response. With the revised algorithm, SeFoRA is best in all three extreme- and standard-rank columns in CIFAR100, Tiny ImageNet, and Food101, in both DomainNet settings in terms of the average accuracy, and on QNLI, and second on SST-2, QQP and MNLI (Table 6,7, and 8).

The classification head is initialized once when constructing the model. All clients start each round from the same current global head, and upload the update without sketching. The local heads are aggregated using the standard FedAvg. Learning rates are selected by sweeping over the candidate values in Tables 4 and 5 in the paper, and choosing the best-performing one. Under partial participation, the selected clients' weights are renormalized over the participating clients. We will add more detailed implementation details in the revised paper.

Table 6. Accuracy (%) on DomainNet using ViT, with rmax⁡=16r_{\max}=16.

αMethodClipartInfographPaintingQuickdrawRealSketchAverage
0.1FlexLoRA63.0 ± 0.531.3 ± 0.461.1 ± 0.332.6 ± 0.278.2 ± 0.253.9 ± 0.553.4 ± 0.3
0.1Fed-PLoRA68.1 ± 0.733.6 ± 0.363.5 ± 0.434.7 ± 0.278.9 ± 0.357.7 ± 0.156.1 ± 0.1
0.1FSLoRA66.7 ± 0.232.9 ± 0.463.4 ± 0.538.3 ± 0.480.4 ± 0.356.9 ± 0.156.4 ± 0.3
0.1FedHL52.1 ± 1.025.3 ± 0.449.6 ± 1.021.4 ± 1.059.9 ± 0.344.3 ± 0.942.1 ± 0.7
0.1SeFoRA67.6 ± 0.233.6 ± 0.263.6 ± 0.340.4 ± 0.480.6 ± 0.257.8 ± 0.257.3 ± 0.1
0.5FlexLoRA63.4 ± 0.232.0 ± 0.361.7 ± 0.433.5 ± 0.178.6 ± 0.354.3 ± 0.353.9 ± 0.1
0.5Fed-PLoRA69.2 ± 0.234.4 ± 0.464.3 ± 0.336.0 ± 0.379.7 ± 0.358.6 ± 0.157.1 ± 0.1
0.5FSLoRA67.5 ± 0.333.4 ± 0.364.1 ± 0.239.2 ± 0.480.8 ± 0.257.5 ± 0.457.1 ± 0.2
0.5FedHL64.0 ± 0.331.1 ± 0.659.2 ± 1.327.5 ± 0.672.0 ± 1.653.3 ± 1.451.2 ± 0.9
0.5SeFoRA67.8 ± 0.133.8 ± 0.164.4 ± 0.141.4 ± 0.280.9 ± 0.258.1 ± 0.157.7 ± 0.1

Table 7. Accuracy (%) on GSM8K and the GLUE tasks, all with rmax⁡=16r_{\max}=16.

MethodGSM8K (Llama)GSM8K (Qwen)SST-2QNLIQQPMNLI
FlexLoRA34.5 ± 0.856.0 ± 0.495.3 ± 0.390.4 ± 1.370.6 ± 12.485.1 ± 2.5
Fed-PLoRA30.1 ± 0.855.3 ± 0.594.1 ± 0.988.4 ± 1.369.5 ± 11.086.3 ± 1.0
FSLoRA36.3 ± 1.155.7 ± 0.895.0 ± 0.190.9 ± 1.170.9 ± 13.587.0 ± 1.0
FedHL21.5 ± 4.655.5 ± 0.793.8 ± 1.682.2 ± 5.975.4 ± 7.777.0 ± 10.6
SeFoRA36.4 ± 0.456.3 ± 0.495.2 ± 0.192.0 ± 1.571.9 ± 11.286.4 ± 2.0

Table 8. Accuracy (%) on CIFAR-100, Tiny ImageNet, and Food-101 under extreme and standard rank heterogeneity.

Rank heterogeneityMethodCIFAR-100Tiny ImageNetFood-101
ExtremeFlexLoRA91.0 ± 0.388.0 ± 0.284.3 ± 0.5
ExtremeFed-PLoRA87.1 ± 0.384.9 ± 0.783.2 ± 1.1
ExtremeFSLoRA88.9 ± 0.386.3 ± 0.283.3 ± 0.9
ExtremeFedHL85.0 ± 0.682.2 ± 0.376.8 ± 3.2
ExtremeSeFoRA91.4 ± 0.288.7 ± 0.186.3 ± 0.4
StandardFlexLoRA91.1 ± 0.287.9 ± 0.184.2 ± 0.6
StandardFed-PLoRA90.8 ± 0.288.2 ± 0.286.6 ± 0.2
StandardFSLoRA91.0 ± 0.188.4 ± 0.286.2 ± 0.8
StandardFedHL83.4 ± 1.279.5 ± 1.375.6 ± 2.9
StandardSeFoRA91.4 ± 0.188.9 ± 0.186.6 ± 0.5
Reviewer_JGTPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SeFoRA, which avoids directly aggregating the LoRA factors. Each client instead communicates linear sketches of its local model-space update. Due to the linearity of the sketching operator, the server can aggregate these sketches directly and reconstruct a low-rank approximation of the aggregated update without explicitly materializing each client’s dense update. The paper also provides a convergence analysis and evaluates the method on language understanding, mathematical reasoning, and vision tasks.

Critical strengths
  1. By aggregating linear sketches of the client model-space updates rather than heterogeneous LoRA factors, SeFoRA naturally supports different client ranks while avoiding direct factor-wise averaging.

  2. The paper provides convergence guarantees and explicitly separates errors due to sketching, local rank truncation, and client heterogeneity.

  3. The evaluation includes RoBERTa-Large on GLUE, 3B language models on GSM8K, and ViT-based experiments on several image classification datasets, including the naturally heterogeneous DomainNet dataset.

Critical weaknesses
  1. The experimental evaluation compares SeFoRA primarily with FlexLoRA, FSLoRA, and FedHL, while omitting recent methods that directly study heterogeneous client LoRA ranks. In particular, Fed-PLoRA (ICLR 2026) explicitly targets federated LoRA under heterogeneous client rank/resource budgets, and FedRoRA (EMNLP 2026) considers federated LoRA under simultaneous rank and data heterogeneity. These methods are highly relevant to the setting studied in this paper. Without comparisons to recent approaches, it is difficult to establish whether SeFoRA provides a better accuracy–communication–computation trade-off than the current state of the art.

  2. SeFoRA generally provides competitive rather than clearly superior predictive performance, so its main advantage appears to be avoiding dense server-side reconstruction while supporting heterogeneous ranks. This makes the practical communication/computation trade-off central to the contribution. However, fixed-size sketches can themselves increase uplink communication for low-rank clients, which the authors acknowledge and address by proposing that such clients instead transmit their LoRA factors and let the server construct the sketches. A more comprehensive empirical comparison of communication volume, server/client computation, memory consumption, and end-to-end training time against recent baselines is needed to establish the practical benefit of the proposed approach.

  3. Across several main experiments, SeFoRA is competitive with rather than consistently better than the strongest baselines. For example, FSLoRA performs better on several GLUE tasks, while FlexLoRA achieves higher accuracy across all three datasets in the extreme-rank experiment. On DomainNet, the improvements over FSLoRA are relatively small. These results may still be acceptable if SeFoRA provides a clearly superior efficiency trade-off, but that trade-off is not sufficiently demonstrated empirically.

  4. Most experiments simulate resource heterogeneity using predefined rank tiers or scaling factors, including the extreme setting with client groups assigned maximum ranks of 8, 16, and 256. While useful as controlled experiments, these settings do not demonstrate how SeFoRA performs when ranks arise from actual heterogeneous device constraints such as memory, computation, or latency budgets. A resource-driven rank allocation experiment would better support the practical motivation.

  5. The key mechanism applies an existing randomized low-rank matrix sketching technique to aggregate federated LoRA updates. The linearity of the sketch naturally enables aggregation before reconstruction, which is useful, but the central technical ingredient itself is not new. Given this relatively straightforward adaptation, stronger empirical evidence of practical advantages over recent heterogeneous-rank FedLoRA approaches would be important.

Questions for authors
  1. How does SeFoRA compare empirically with recent heterogeneous-rank federated LoRA methods such as Fed-PLoRA and FedRoRA, particularly in terms of accuracy, communication, server computation, and memory?
  2. Since the primary advantage over FlexLoRA appears to be avoiding dense server-side reconstruction, can the authors provide end-to-end wall-clock, peak server memory, and actual communication-volume comparisons across different model and rank configurations?
  3. How does the communication advantage change as the ratio between the smallest and largest client ranks increases? In particular, when r_i \ll r_{\max}, under what regimes is transmitting the sketch still preferable to transmitting the client’s LoRA factors?
4 comments

We thank Reviewer JGTP for their detailed review and insightful comments. Before answering the reviewer's questions in detail, we note that a comment from Reviewer 683P led to a change in the method: the clients now sketch their final local adapters Bi(t,E)Ai(t,E)\mathbf{B}_i^{(t,E)}\mathbf{A}_i^{(t,E)} directly, rather than using the relative change Bi(t,E)Ai(t,E)−Bi(t,0)Ai(t,0)\mathbf{B}_i^{(t,E)}\mathbf{A}_i^{(t,E)}-\mathbf{B}_i^{(t,0)}\mathbf{A}_i^{(t,0)}. The revised aggregation rule allows SeFoRA to consistently outperform other methods, especially in extreme settings.

W1/Q1: Comparison with Fed-PLoRA and FedRoRA

Response. Thank you for pointing out these two related works. We include the comparison with Fed-PLoRA below and will add it as a closely related work in the manuscript. As shown in the updated tables below (which will be updated in the manuscript), Fed-PLoRA is competitive, tying for second in average accuracy at α=0.5\alpha=0.5, but performs poorly on the quickdraw domain. It is weaker under extreme rank heterogeneity and on GSM8K with Llama. Its Select-N-Fold method requires the full adapters on each client, so every client downloads the full rank-rr adapter, increasing downlink communication costs.

FedRoRA learns personalized adapters and evaluates personalized models, whereas SeFoRA and all our baselines concern a single global model. We will include the paper in the related work section of Appendix D, and clarify this distinction.

Table 1. Accuracy (%) on DomainNet using ViT, with rmax⁡=16r_{\max}=16.

αMethodClipartInfographPaintingQuickdrawRealSketchAverage
0.1FlexLoRA63.0 ± 0.531.3 ± 0.461.1 ± 0.332.6 ± 0.278.2 ± 0.253.9 ± 0.553.4 ± 0.3
0.1Fed-PLoRA68.1 ± 0.733.6 ± 0.363.5 ± 0.434.7 ± 0.278.9 ± 0.357.7 ± 0.156.1 ± 0.1
0.1FSLoRA66.7 ± 0.232.9 ± 0.463.4 ± 0.538.3 ± 0.480.4 ± 0.356.9 ± 0.156.4 ± 0.3
0.1FedHL52.1 ± 1.025.3 ± 0.449.6 ± 1.021.4 ± 1.059.9 ± 0.344.3 ± 0.942.1 ± 0.7
0.1SeFoRA67.6 ± 0.233.6 ± 0.263.6 ± 0.340.4 ± 0.480.6 ± 0.257.8 ± 0.257.3 ± 0.1
0.5FlexLoRA63.4 ± 0.232.0 ± 0.361.7 ± 0.433.5 ± 0.178.6 ± 0.354.3 ± 0.353.9 ± 0.1
0.5Fed-PLoRA69.2 ± 0.234.4 ± 0.464.3 ± 0.336.0 ± 0.379.7 ± 0.358.6 ± 0.157.1 ± 0.1
0.5FSLoRA67.5 ± 0.333.4 ± 0.364.1 ± 0.239.2 ± 0.480.8 ± 0.257.5 ± 0.457.1 ± 0.2
0.5FedHL64.0 ± 0.331.1 ± 0.659.2 ± 1.327.5 ± 0.672.0 ± 1.653.3 ± 1.451.2 ± 0.9
0.5SeFoRA67.8 ± 0.133.8 ± 0.164.4 ± 0.141.4 ± 0.280.9 ± 0.258.1 ± 0.157.7 ± 0.1

Table 2. Accuracy (%) on GSM8K and the GLUE tasks, all with rmax⁡=16r_{\max}=16.

MethodGSM8K (Llama)GSM8K (Qwen)SST-2QNLIQQPMNLI
FlexLoRA34.5 ± 0.856.0 ± 0.495.3 ± 0.390.4 ± 1.370.6 ± 12.485.1 ± 2.5
Fed-PLoRA30.1 ± 0.855.3 ± 0.594.1 ± 0.988.4 ± 1.369.5 ± 11.086.3 ± 1.0
FSLoRA36.3 ± 1.155.7 ± 0.895.0 ± 0.190.9 ± 1.170.9 ± 13.587.0 ± 1.0
FedHL21.5 ± 4.655.5 ± 0.793.8 ± 1.682.2 ± 5.975.4 ± 7.777.0 ± 10.6
SeFoRA36.4 ± 0.456.3 ± 0.495.2 ± 0.192.0 ± 1.571.9 ± 11.286.4 ± 2.0

Table 3. Accuracy (%) on CIFAR-100, Tiny ImageNet, and Food-101 under extreme and standard rank heterogeneity.

Rank heterogeneityMethodCIFAR-100Tiny ImageNetFood-101
ExtremeFlexLoRA91.0 ± 0.388.0 ± 0.284.3 ± 0.5
ExtremeFed-PLoRA87.1 ± 0.384.9 ± 0.783.2 ± 1.1
ExtremeFSLoRA88.9 ± 0.386.3 ± 0.283.3 ± 0.9
ExtremeFedHL85.0 ± 0.682.2 ± 0.376.8 ± 3.2
ExtremeSeFoRA91.4 ± 0.288.7 ± 0.186.3 ± 0.4
StandardFlexLoRA91.1 ± 0.287.9 ± 0.184.2 ± 0.6
StandardFed-PLoRA90.8 ± 0.288.2 ± 0.286.6 ± 0.2
StandardFSLoRA91.0 ± 0.188.4 ± 0.286.2 ± 0.8
StandardFedHL83.4 ± 1.279.5 ± 1.375.6 ± 2.9
StandardSeFoRA91.4 ± 0.188.9 ± 0.186.6 ± 0.5

W2: SeFoRA generally provides competitive rather than clearly superior predictive performance, so its main advantage appears to be avoiding dense server-side reconstruction while supporting heterogeneous ranks. This makes the practical communication/computation trade-off central to the contribution. However, fixed-size sketches can themselves increase uplink communication for low-rank clients, which the authors acknowledge and address by proposing that such clients instead transmit their LoRA factors and let the server construct the sketches. A more comprehensive empirical comparison of communication volume, server/client computation, memory consumption, and end-to-end training time against recent baselines is needed to establish the practical benefit of the proposed approach.

Q2: Since the primary advantage over FlexLoRA appears to be avoiding dense server-side reconstruction, can the authors provide end-to-end wall-clock, peak server memory, and actual communication-volume comparisons across different model and rank configurations?

By slightly modifying the final update and eliminating the error feedback, SeFoRA now consistently outperforms FlexLoRA. Nevertheless, we provide an end-to-end wall clock and memory measurements on CIFAR-100 using ViT-base as the model, under standard heterogeneous ranks with maximum rank 1616 and extreme heterogeneous ranks with tiers 8, 16 and 256.

Table 4 shows that, with sketch upload, SeFoRA reduces the federator time per round and the federator peak memory compared with FlexLoRA, at the expense of increased communication. The total time that 30 clients spend per round on sketching is relatively light compared with the local training time. In addition, the revised SeFoRA reaches higher accuracy than FlexLoRA in both standard and extreme settings. So SeFoRA with sketch upload outperforms FlexLoRA in terms of accuracy, computation, and memory.

In terms of communication, uploading sketches increases communication cost, especially when the clients' rank heterogeneity is extreme, as shown in Table 5. Thus, in the extreme setting, clients with ri,l≪rlr_{i,l} \ll r_l may upload their factors to the federator. Factor upload reduces uplink communication by transferring sketch construction from clients to the federator. Note that the table only reports a uniform upload policy, i.e., clients either all upload factors or sketches. A mixed policy can be calculated by summing the corresponding payload for each tier. Since the sketch is linear and Ω\Omega and Ψ\Psi are shared, computing the sketch on the server or the client would yield identical accuracy.

Table 4. Per-round computation time and server memory measurements on CIFAR-100, with all clients uploading their sketches in SeFoRA. Server time includes aggregation and construction of the next global adapters. Compute time is the sum of server time, total client sketching time, and total client training time. The standard and extreme settings were run on different GPUs (RTX PRO 4500 and RTX PRO 5000), so compare methods only within each setting.

SettingMethodAccuracy (%)Server (s/round)Total client sketch (s/round)Total client train (s/round)Compute (s/round)Server peak (MiB)
StandardFlexLoRA91.1 ± 0.20.21206.486.69286.8
StandardSeFoRA91.4 ± 0.10.0200.0586.516.59240.1
ExtremeFlexLoRA91.0 ± 0.30.88905.116.00411.8
ExtremeSeFoRA91.4 ± 0.20.1860.2005.125.51270.8

Table 5. Accuracy and computed communication (MiB) per round on CIFAR-100 (all 30 clients, adapters and head). Standard: rmax⁡=16r_{\max}=16; extreme: rmax⁡=256r_{\max}=256.

MethodStandard accuracy (%)Standard up / down (MiB)Standard two-way (MiB)Extreme accuracy (%)Extreme up / down (MiB)Extreme two-way (MiB)
FlexLoRA91.1 ± 0.242.8 / 42.885.691.0 ± 0.3299.7 / 299.7599.4
Fed-PLoRA90.8 ± 0.242.8 / 55.097.887.1 ± 0.3299.7 / 814.41114.1
FSLoRA91.0 ± 0.142.8 / 55.097.888.9 ± 0.3299.7 / 814.41114.1
FedHL83.4 ± 1.242.8 / 42.885.685.0 ± 0.6299.7 / 299.7599.4
SeFoRA, factor upload91.4 ± 0.142.8 / 42.885.691.4 ± 0.2299.7 / 299.7599.4
SeFoRA, sketch upload91.4 ± 0.159.2 / 42.8102.091.4 ± 0.2865.9 / 299.71165.6

W3: Across several main experiments, SeFoRA is competitive with rather than consistently better than the strongest baselines. For example, FSLoRA performs better on several GLUE tasks, while FlexLoRA achieves higher accuracy across all three datasets in the extreme-rank experiment. On DomainNet, the improvements over FSLoRA are relatively small. These results may still be acceptable if SeFoRA provides a clearly superior efficiency trade-off, but that trade-off is not sufficiently demonstrated empirically.

W5: The key mechanism applies an existing randomized low-rank matrix sketching technique to aggregate federated LoRA updates. The linearity of the sketch naturally enables aggregation before reconstruction, which is useful, but the central technical ingredient itself is not new. Given this relatively straightforward adaptation, stronger empirical evidence of practical advantages over recent heterogeneous-rank FedLoRA approaches would be important.

Response. The main novelty of the paper does not lie in the construction of the sketching operator, which we borrow from Tropp et al.\ (2017), rather in enabling rank-heterogeneous federated LoRA with linear operations and showing convergence theoretically and through extensive numerical experiments. Because the sketch is linear, the sum of the clients' fixed-size sketches equals the sketch of the exact aggregate ∑ipiBiAi\sum_i p_i\mathbf B_i\mathbf A_i. This avoids the bilinear mismatch and the construction of the dense update in the full space. We also give a convergence analysis that separates the sketching, truncation, and heterogeneity errors.

With the new rule that drops error compensation, SeFoRA outperforms other methods in most experiments, particularly under extreme rank heterogeneity and on DomainNet with α=0.1\alpha=0.1. To enrich our empirical evidence, we add more experiments, as detailed in the response to W2/Q2, to better present the arising trade-offs.

W4: Most experiments simulate resource heterogeneity using predefined rank tiers or scaling factors, including the extreme setting with client groups assigned maximum ranks of 8, 16, and 256. While useful as controlled experiments, these settings do not demonstrate how SeFoRA performs when ranks arise from actual heterogeneous device constraints such as memory, computation, or latency budgets. A resource-driven rank allocation experiment would better support the practical motivation.

Response. To simulate the clients' actual heterogeneous device constraints, we apply client-specific rank profiles on all methods. This controlled setup isolates how aggregation methods handle a given distribution of clients' device constraints. While predefined rank profiles do not demonstrate or validate a mapping from device constraints such as memory, computation, or latency budgets to ranks, SeFoRA takes any local rank profile as input and is agnostic to how those profiles are computed. Choosing the profile from memory, compute, or latency budgets is a resource-allocation problem that is complementary to aggregation and outside the scope of this paper (footnote 2 of the paper), and the chosen allocator can be plugged in unchanged.

Q3: How does the communication advantage change as the ratio between the smallest and largest client ranks increases? In particular, when ri≪rmax⁡r_i \ll r_{\max}, under what regimes is transmitting the sketch still preferable to transmitting the client’s LoRA factors?

Response. For each layer, the uplink is dr+skdr+sk for transmitting the sketch, and is (d+s)ri(d+s)r_i for transmitting the clients' LoRA factors. Since ri≤r<kr_i\le r<k, the factors are always smaller, and the ratio grows linearly in r/rir/r_i. The relation ri,l≪rlr_{i,l}\ll r_l was intended for the clients most affected by the fixed sketch size. The option of uploading factors offers particularly large relative communication savings for low-rank clients. When ri,lr_{i,l} is close to rlr_l, sketch upload incurs only a modest relative overhead. However, factor upload transfers computation to the federator and reveals the non-linear LoRA factors to it. If the federator computation is limited, or the federator must not see individual adapters, sketch upload is preferable.

We hope that our explanation and the additional numerical experiments address the reviewer's concerns, and we are happy to provide further details.

Dear Reviewer JGTP,

We wanted to briefly follow up on our response and revised manuscript. The revision expands comparisons with related methods, reports results for the refined algorithm, which outperforms the baselines, and includes additional system measurements. We hope these updates and our responses address your concerns. Please let us know if you have any further questions. We are happy to discuss further.

Best regards, The Authors

Comments on the paper

2 comments

We thank the Reviewers and the Area Chair for their time and thoughtful reviews. We have updated the revised manuscript to address the Reviewers' constructive feedback (all changes are highlighted in blue). Below, we summarize the main revisions.

  1. Slight revision of the algorithm. Motivated by Reviewer 683P's comments, we dropped the error compensation component in aggregation, i.e., the clients now sketch their final local adapters Bi(t,E)Ai(t,E)\mathbf{B}_i^{(t,E)}\mathbf{A}_i^{(t,E)} directly, rather than using the relative change Bi(t,E)Ai(t,E)−Bi(t,0)Ai(t,0)\mathbf{B}_i^{(t,E)}\mathbf{A}_i^{(t,E)}-\mathbf{B}_i^{(t,0)}\mathbf{A}_i^{(t,0)} (Section 3, p.5; Appendix A, p.18). We rerun all SeFoRA entries in the main result using the same data splits, rank profiles, rounds, and seeds.

  2. Expanded comparisons and updated results. We added the closely related paper suggested by Reviewer JGTP as a baseline (Section 1, p.2; Section 5, p.8-9). We also discussed FedRoRA and clarified that it focuses on personalized models, whereas SeFoRA learns a single global model (Appendix D, p. 25). The revised SeFoRA achieves the highest average accuracy in both DomainNet settings, across all three vision datasets, and on both GSM8K models in both standard and extreme rank-heterogeneity settings, and remains competitive on the GLUE benchmark (Section 5, p.8-9; Appendix C, p.21).

  3. Updated convergence analysis and supporting experiments. We revised Theorem 1 and its proofs to cover the revised aggregation rule (Section 4, p.6-7; Appendix E). The bound now contains one additional truncation error term Etrunc\mathcal{E}_{\text{trunc}}. Corollary 1 is restated as an O(T−1/2)\mathcal{O}(T^{-1/2}) transient plus an explicit limiting neighborhood. To connect the SGD-based analysis with the empirical evaluation, as noted by Reviewer QKTi, we added matched SGD experiments across all three vision datasets under both standard and extreme rank heterogeneity (Figure 2, Appendix C, p.21). We also report spectral-tail diagnostics that provide evidence relevant to Assumption 5 (Appendix C, p.22).

  4. Clarified upload policy. All reported experiments use client-side sketching, i.e., every client computes and uploads its sketches (Section 5, p.9). Uploading LoRA factors is an alternative for bandwidth-constrained clients, particularly those with rank ri,l≪rlr_{i,l}\ll r_l. With shared sketching matrices Ω\Omega and Ψ\Psi, both choices produce the same aggregated sketches and the global adapters, but demonstrate a trade-off between communication and computation.

  5. More complete efficiency measurements. We report the system measurements including server processing time, clients computation time, communication payloads, and memory usage (Appendix C, p. 24). These measurements make the communication-computation trade-off explicit: client-side sketching reduces server computation while increasing uplink communication.

  6. Secure-aggregation compatibility. As privacy is not the main concern in this work, we assume that the federator may access individual client factors; thus, clients can shift sketch computation to the federator. However, if privacy-preserving aggregation is required, either the clients must compute the sketches, so that secure aggregation is directly compatible, increasing communication costs, or additional protection is required.

  7. Additional implementation details. We clarify classification-head initialization and aggregation, and aggregation weights under partial participation (Appendix B, p.17).

AuthorsOpenReview ↗

Author summary of the rebuttal

Dear Area Chair,

Summary of changes

We would like to provide a brief overview of the rebuttal so far. We posted detailed responses to all three reviewers (JGTP, QKTi, and 683P) and revised the manuscript in light of their feedback. The revised manuscript includes an updated SeFoRA algorithm, updated experiments, additional ablation studies, new baselines, a justification of Assumption 5, and updated theoretical results that match our new algorithm. All changes in the manuscript are highlighted in blue for convenience.

We believe these revisions and our responses address the concerns raised by the reviewers. We respectfully ask that you consider the completed revisions, additional evidence, and discussion alongside the original reviews when assessing the submission.

Updated algorithm

We slightly modified the algorithm by removing the change-relative term (error compensation) based on a comment from Reviewer 683P. We modified the theoretical analysis and reran all the experiments for this version. We now show that SeFoRA has superior performance compared to state of the art as we detail below.

Improving over state-of-the-art

SeFoRA achieves the highest average accuracy on DomainNet under both label heterogeneity settings (α=\alpha= 0.1 and 0.5), and the highest mean accuracy on both GSM8K models. It ranks first across all three vision datasets under both standard and extreme rank-heterogeneity settings. On GLUE, the strongest baseline varies across tasks, while SeFoRA consistently ranks first or second, i.e., first on QNLI, and second on SST-2, QQP, and MNLI. Those results are reported in Tables 1--3 in our response to Reviewer JGTP (and are replicated in our responses to the other reviewers and in the updated manuscript).

Based on a comment from Reviewer JGTP, we added Fed-PLoRA as a baseline in all our experiments and clarified the difference between the personalized FedRoRA and the setting we consider.

System measurements and tradeoffs

We added an experiment that shows the system measurements for communication, computation, and memory. This experiment highlights the trade-offs that arise from using SeFoRA. The results are reported in Tables 3--5 in our response to Reviewer QKTi (and are replicated in the updated manuscript).

Added ablation

We added experiments comparing the different SeFoRA variants with FlexLoRA, as suggested by Reviewer 683P. The experiments highlight the difference between FlexLoRA and SeFoRA and suggest removing the change-relative term, which we ultimately did.

Verifying additional assumptions and running SGD (not Adam)

Since LoRA is based on the assumption that the update consists of a low-rank matrix, we made Assumption 5 in the manuscript bounding the spectral tail of the matrices. Based on a comment from Reviewer QKTi, we verified this assumption through additional numerical experiments reported in Appendix C of the updated manuscript.

Since our theory considers SGD, we ran experiments that match it and show that SeFoRA converges, albeit more slowly, in this setting as well. For a fair comparison with the state of the art (which does a similar thing), we kept all other experiments using Adam.

Reviewer interaction

Reviewer QKTi stated that our response essentially addresses their concerns about convergence analysis, communication and computational costs, and the upload rule, and updated their score to 3.

Reviewer 683P replied to our first rebuttal and raised concerns about experimental scope and the gains over FSLoRA before we had the chance to update the manuscript. We responded with an updated manuscript and the complete experiments, and highlighted clear gains of 2.4-3.0 percentage points over FSLoRA in the extreme rank heterogeneity setting.

Reviewer JGTP has not yet responded to our rebuttal; we have provided point-by-point responses and followed up to invite further discussion.

Thank you

We thank you in advance for the time you are investing in this review process and appreciate your consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.