- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper looks at federated LoRA when clients use different adapter ranks. Instead of zero-padding every client to , NSVD-Agg keeps each client update at its native rank and right-projects it onto a shared basis, .
The server then aggregates the projected products. The basis is the top- eigenspace of computed from the raw updates of the previous round and kept fixed during the current round. The authors argue that this one-round lag makes measurable with respect to and that their non-convex bound requires it:
The experiments use Llama-3.2-3B and Qwen2.5-3B on tiny-Shakespeare, with 20 Dirichlet clients and 10 rounds. Ranks are simulated by truncating rank-32 updates. NSVD-Agg beats zero-padding in all 7 seeds on both models, by about to in loss.
The authors' own ablation in Section 5.8 shows that an untracked "exact-product" baseline, which takes the exact weighted aggregate and truncates it to rank , matches or beats NSVD-Agg in every seed. The paper therefore reframes tracking as the price paid for the convergence guarantee.
Critical strengths
The paper studies an important problem in federated LoRA aggregation with heterogeneous ranks and is honest about negative results, as the ablation study in Section 5.8 shows that the gain over zero-padding basically comes from exact-product aggregation. The negative findings on (no decay) and the shrinking held-out advantage in B.4 are reported in the same spirit. I value this, and it is the main reason I engaged closely with the paper.
Critical weaknesses
1. The guarantee does not need the lag, so it does not separate NSVD-Agg from the exact-product baseline
The paper's main argument is that tracking "supplies the guarantee" that the exact-product baseline lacks, because only NSVD-Agg's is -measurable. This claim appears many times in the paper, including the abstract and conclusion. However, I could not find where measurability is used in the proof:
- In Lemma 3, the cross-term vanishes because , and is defined before projection. It does not involve at all.
- The cross-term is handled with Young's inequality, which needs no measurability.
- Lemma 2 only uses , which holds for any orthogonal projector, even if computed from the current round's data.
So the same theorem holds for the exact-product operator. Its error is bounded by the same quantity. In fact, by Eckart-Young, the best rank- truncation of the aggregate has the smallest aggregate error of any rank- projection. If this is right, the paper's final position, "a proven guarantee at a small, measured cost", does not hold. The simpler, better-performing baseline has the same guarantee.
2. The bound is also weak in its own right:
Under A3, , so the deterministic bound yields . The theorem therefore gives convergence to an neighborhood, not to a stationary point. This is the standard result for SGD with a biased gradient oracle, where a bias of magnitude gives an floor (Ajalloeian & Stich, 2020). LBGM (Azam et al., ICLR 2022) already has the federated analog with a projection-error floor. Neither is cited.
3. Exact-product aggregation with SVD re-factorization under heterogeneous ranks.
FlexLoRA (Bai et al., NeurIPS 2024) already does this, and the paper cites it. The "untracked exact-product baseline" is essentially FlexLoRA with a single rank cap. It should be named as such rather than introduced as a new baseline. Section 5.8's main finding, that exact products rather than tracking explain the gain, is therefore a rediscovery.
FedEx-LoRA (Singhal et al., 2025) is cited, but it is described as "orthogonal", even though its exactness argument is precisely what Section 5.8 ends up attributing the gain to.
4. The communication saving is not specific to the method.
Padding can be done on the server, so HetLoRA as published, FlexLoRA, HLoRA, FLoRA, and the exact-product baseline all upload at native rank and get the same . Figure 1 suggests the client uploads the projected update, which would require broadcasting each round. That cost is excluded at L346-347. The download path is never described: how the aggregate, of rank at most , reaches the rank- clients and how it is merged into the NF4 base.
5. The evaluation is small
The "held-out" set consists of tokens 1280-1536 from the first five clients only, totaling 1,280 tokens from 5 of the 20 clients. With five clients per rank, all of them may have the same rank. The in-training-window loss is higher than the held-out loss (3.8665 vs. 3.8208 for NSVD-Agg), which is unexpected and unexplained.
6. The step-size corollary is inconsistent.
For fixed , we have already forces , so . The suggested (L664) contradicts this. Also, is a constant for a fixed horizon, not a rate that "shrinks across rounds" (L665).
Questions for authors
Please see the weaknesses.