acceptodds
Under review as a conference paper at ICLR 2027

Beyond Zero-Padding: Nested-SVD Aggregation for Heterogeneous-Rank Federated LoRA

Abstract

Federated fine-tuning with LoRA typically assumes all clients share one adapter rank, even though heterogeneous devices may need different ranks to meet local memory and compute constraints. The common remedy, zero-padding smaller LoRA factors to a shared maximum rank, preserves a uniform parameter shape but communicates the update in a larger shape than it needs. We propose NSVD-Agg, a nested-SVD aggregation method that instead projects each client’s native-rank update onto a shared reference subspace of dimension \(r_\max\). The subspace is estimated from the raw client updates and refreshed across rounds, while the current basis stays fixed during aggregation – a temporal ordering required for the non-convex convergence guarantee we prove. On Llama-3.2-3B with simulated heterogeneous ranks \({4,8,16,32}\), NSVD-Agg achieves lower final loss than zero-padding in all seven random seeds while reducing client-to-server LoRA communication by an analytical \(53.1%\); the comparison replicates on Qwen2.5-3B, where the improvement is larger and more consistent. We further introduce an untracked exact-product baseline that shares NSVD-Agg’s communication savings but, by construction, lacks its guarantee; across seven seeds on both models it matches or slightly exceeds NSVD-Agg’s loss, indicating the gain over zero-padding comes primarily from exact-product aggregation rather than from tracking itself. Tracking’s role is therefore to supply a proven convergence guarantee at a small, measured empirical cost. NSVD-Agg is thus a communication-efficient, theoretically grounded aggregation scheme whose empirical advantage and theoretical guarantee arise from distinct, separately-verified mechanisms.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Federated Learning, LoRA, Parameter-Efficient Fine-Tuning, Heterogeneous-Rank Aggregation, Federated Fine-Tuning, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YQ48OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at federated LoRA when clients use different adapter ranks. Instead of zero-padding every client to rmax⁡r_{\max}, NSVD-Agg keeps each client update Δwk,t=Ak,tBk,t⊤\Delta w_{k,t} = A_{k,t}B_{k,t}^{\top} at its native rank and right-projects it onto a shared basis, Δw~k,t=Δwk,tPt\widetilde{\Delta w}_{k,t}=\Delta w_{k,t}P_t.

The server then aggregates the projected products. The basis is the top-rmax⁡r_{\max} eigenspace of Ct=∑kpkΔwk,t⊤Δwk,t,C_t = \sum_k p_k \Delta w_{k,t}^{\top}\Delta w_{k,t}, computed from the raw updates of the previous round and kept fixed during the current round. The authors argue that this one-round lag makes PtP_t measurable with respect to Ft\mathcal{F}_t and that their non-convex bound requires it:

min⁡tE[∥∇f(wt)∥2]=O(1T+DT+RT).\min_t \mathbb{E}\left[\lVert \nabla f(w_t)\rVert^2\right] = O\left(\frac{1}{\sqrt{T}} + D_T + R_T\right).

The experiments use Llama-3.2-3B and Qwen2.5-3B on tiny-Shakespeare, with 20 Dirichlet clients and 10 rounds. Ranks 4,8,16,32\\{4,8,16,32\\} are simulated by truncating rank-32 updates. NSVD-Agg beats zero-padding in all 7 seeds on both models, by about 0.020.02 to 0.030.03 in loss.

The authors' own ablation in Section 5.8 shows that an untracked "exact-product" baseline, which takes the exact weighted aggregate and truncates it to rank rmax⁡r_{\max}, matches or beats NSVD-Agg in every seed. The paper therefore reframes tracking as the price paid for the convergence guarantee.

Critical strengths

The paper studies an important problem in federated LoRA aggregation with heterogeneous ranks and is honest about negative results, as the ablation study in Section 5.8 shows that the gain over zero-padding basically comes from exact-product aggregation. The negative findings on αk,t\alpha_{k,t} (no decay) and the shrinking held-out advantage in B.4 are reported in the same spirit. I value this, and it is the main reason I engaged closely with the paper.

Critical weaknesses

1. The guarantee does not need the lag, so it does not separate NSVD-Agg from the exact-product baseline

The paper's main argument is that tracking "supplies the guarantee" that the exact-product baseline lacks, because only NSVD-Agg's PtP_t is Ft\mathcal{F}_t-measurable. This claim appears many times in the paper, including the abstract and conclusion. However, I could not find where measurability is used in the proof:

  • In Lemma 3, the ξt\xi_t cross-term vanishes because E[ξt∣Ft]=0\mathbb{E}[\xi_t \mid \mathcal{F}_t] = 0, and ξt\xi_t is defined before projection. It does not involve PtP_t at all.
  • The etranke^{\mathrm{rank}}_t cross-term is handled with Young's inequality, which needs no measurability.
  • Lemma 2 only uses ∥I−Pt∥2≤1\|I - P_t\|_2 \le 1, which holds for any orthogonal projector, even if computed from the current round's data.

So the same theorem holds for the exact-product operator. Its error 1η∥Δwt(I−Pt′)∥\frac{1}{\eta}\|\Delta w_t (I - P_t')\| is bounded by the same quantity. In fact, by Eckart-Young, the best rank-rmax⁡r_{\max} truncation of the aggregate has the smallest aggregate error of any rank-rmax⁡r_{\max} projection. If this is right, the paper's final position, "a proven guarantee at a small, measured cost", does not hold. The simpler, better-performing baseline has the same guarantee.


2. The bound is also weak in its own right:

Under A3, ∣Δwk,t∣F/η=∥gˉk,t∥≤G|\Delta w_{k,t}|_F/\eta = \|\bar g_{k,t}\| \le G, so the deterministic bound yields RT≤G2R_T \le G^2. The theorem therefore gives convergence to an O(G2)O(G^2) neighborhood, not to a stationary point. This is the standard result for SGD with a biased gradient oracle, where a bias of magnitude ζ\zeta gives an O(ζ2)O(\zeta^2) floor (Ajalloeian & Stich, 2020). LBGM (Azam et al., ICLR 2022) already has the federated analog with a projection-error floor. Neither is cited.


3. Exact-product aggregation with SVD re-factorization under heterogeneous ranks.

FlexLoRA (Bai et al., NeurIPS 2024) already does this, and the paper cites it. The "untracked exact-product baseline" is essentially FlexLoRA with a single rank cap. It should be named as such rather than introduced as a new baseline. Section 5.8's main finding, that exact products rather than tracking explain the gain, is therefore a rediscovery.

FedEx-LoRA (Singhal et al., 2025) is cited, but it is described as "orthogonal", even though its exactness argument is precisely what Section 5.8 ends up attributing the gain to.


4. The communication saving is not specific to the method.

Padding can be done on the server, so HetLoRA as published, FlexLoRA, HLoRA, FLoRA, and the exact-product baseline all upload at native rank and get the same 1−15/32=53.1%1 - 15/32 = 53.1\%. Figure 1 suggests the client uploads the projected update, which would require broadcasting Vt⋆∈Rd×rmax⁡V_t^\star \in \mathbb{R}^{d \times r_{\max}} each round. That cost is excluded at L346-347. The download path is never described: how the aggregate, of rank at most rmax⁡r_{\max}, reaches the rank-rkr_k clients and how it is merged into the NF4 base.


5. The evaluation is small

The "held-out" set consists of tokens 1280-1536 from the first five clients only, totaling 1,280 tokens from 5 of the 20 clients. With five clients per rank, all of them may have the same rank. The in-training-window loss is higher than the held-out loss (3.8665 vs. 3.8208 for NSVD-Agg), which is unexpected and unexplained.


6. The step-size corollary is inconsistent.

For fixed τ\tau, we have η=ηℓτ=Θ(T−1/2)\eta = \eta_\ell \tau = \Theta(T^{-1/2}) already forces ηℓ=Θ(T−1/2)\eta_\ell = \Theta(T^{-1/2}), so DT=O(1/T)D_T = O(1/T). The suggested ηℓ=O(T−1/4)\eta_\ell = O(T^{-1/4}) (L664) contradicts this. Also, Θ(T−1/2)\Theta(T^{-1/2}) is a constant for a fixed horizon, not a rate that "shrinks across rounds" (L665).

Questions for authors

Please see the weaknesses.

Reviewer_VSnEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles rank heterogeneity in federated LoRA. When clients have different ranks, the resulting adapters have different dimensions and cannot be immediately aggregated.

There is a large body of literature working on this topic, some of which this paper is missing. The literature that this paper compares to zero-pads the factors to make them addition-compatible. The paper operates under the premise that zero-padding increases the communication cost and that it can provide a better solution.

To that end, the paper proposes a solution that requires each client to project its update onto the same subspace specified by the federator. This projection makes the factors have the same dimensions.

In my opinion, the paper's claims are not sustained. Please see the discussion below.

Critical strengths

If my concerns hold, this paper does not improve the state of the art.

Critical weaknesses

The paper has several critical weaknesses:

  1. The paper assumes that methods using zero-padding require the clients to transmit the zeros. This assumption is the main motivation behind the claimed savings in communication. However, this assumption is not true. The clients need not transmit the zeros.

  2. The explanation in text and Figure 1 are misleading. Figure 1 shows that the client's ranks after projection retain the same dimension. However, on closer inspection, the projection matrix PtP_t has dimension d×dd\times d. So projecting on that matrix does not change any dimension. Besides, it seems that in the proposed algorithm, the clients are transmitting ΔW\Delta W, which means transmitting d2d^2 entries rather than 2d r_\max. Therefore, it is not clear where the gains come from.

  3. Suppose there is a gain in communication that I missed. The federator still needs to compute Pt+1P_{t+1} which requires observing the raw updates of the clients. How is that done?

  4. The paper restricts its comparison to a dumbed-down version of HetLoRA (omitting the aggregation weights). However, upon reading HetLoRA closely, the authors show that the aggregation weights are the main reason zero-padding may work. Comparing to HetLoRA without those weights is therefore not a fair comparison.

  5. Why does the paper not compare against FLoRA, FlexLoRA, FedHL and the paper by (Waseem et al.)? The paper also misses a large body of the literature using similar ideas, e.g., FedRot-LoRA and FSLoRA

FedRot-LoRA: Zhang, Haoran, FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA, ICML 2026.

FSLoRA: Fang et al. Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs, ICML 2026.

  1. The introduced method is restricted to factors AA and BB having the same dimension, which is not the case for any of the methods in the literature. This not only limits applicability, but requires explaining how the method deals with the fact that the key/value matrices remain fixed.

  2. In Section 5.1, the paper explains that the clients first train updates at the uniform maximum capacity, then truncate the updates to the assigned ranks. This is different from clients training adapters with different ranks from the beginning, as commonly considered in the literature. In practice, clients adopt heterogeneous ranks due to differences in device memory and computational constraints (also mentioned in the early sections of this paper).

  3. In section 5.8, the paper compares with the exact-product approach, i.e., an exact-product aggregation followed by rank r_\max SVD approximation. Table 2 reports lower loss for exact-product aggregation. However, the text states that the exact-product variant “matches or slightly exceeds NSVD-Agg’s loss”, which contradicts the table and should be clarified. More importantly, as the authors acknowledge, the improvement over zero-padding is due to product aggregation rather than tracking, leaving the benefit of the proposed tracking mechanism unclear.

Questions for authors

Please see the critical weaknesses above.

Reviewer_H68POpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies aggregation of heterogeneous-rank LoRA updates in federated fine-tuning. NSVD-Agg uploads native-rank factors, aggregates their products after right projection onto a shared subspace, and updates that subspace from the previous round's raw client updates. On tiny-Shakespeare with two 3B models and seven seeds, the method improves on separate-factor averaging with zero-padding.

Critical strengths

The paper addresses a relevant problem in federated LoRA fine-tuning and presents a clear aggregation scheme based on a shared reference subspace.

Critical weaknesses
  1. The main empirical comparison is against Zero-pad, without direct comparisons to representative methods for heterogeneous-rank federated LoRA, such as FlexLoRA, HetLoRA, and FLoRA. The current results therefore do not establish an advantage over existing approaches.
  2. The evaluation uses only tiny-Shakespeare. Although two model families are tested, the use of a single small corpus provides limited evidence for the method's effectiveness and generalizability. The reported reductions in evaluation loss over Zero-pad are also small in absolute terms, making the practical benefit difficult to assess from the current experiments.
  3. All clients train at the same maximum rank, and their updates are truncated to the assigned ranks only before upload. This does not reflect the stated motivation that heterogeneous devices need different ranks to meet local memory and compute constraints: lower-rank clients still incur the training cost of the maximum-rank adapter. The experiments therefore do not validate the method under actual training-time resource heterogeneity.
  4. LoRA fine-tuning is restricted to the query and output projections in the attention layers, andThe paper explicitly states that the method does not support non-square modules.. This restriction substantially narrows the scope of the method and its empirical findings.
Questions for authors
  1. Why are there no direct comparisons with FlexLoRA, HetLoRA, and FLoRA, or with the other relevant methods mentioned in the limitations, including RBLA and FedHL?
  2. Truncating an update after maximum-rank local training may discard information learned during optimization. How much does this truncation affect the final performance, and could its effect outweigh the differences between aggregation methods?
  3. The paper states that non-square modules are currently unsupported, but does not explain the specific mathematical or implementation constraint. Which step of NSVD-Agg requires the update matrix to be square, and why can that step not be applied to a rectangular matrix?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.