acceptodds
Under review as a conference paper at ICLR 2027

Outlier Capping for One-shot Federated LoRA: Dominated, Not Misaligned

Abstract

One-shot federated merging combines LoRA adapters that clients trained independently into one adapter, in a single round and without data, gradients, or a validation set. In this setting recent merge operators collapse, and on eight financial adapters for Llama-3.1-8B and hub adapters for Mistral-7B several of them score exactly zero. Realigning client subspaces is a natural remedy, but we show that misalignment is not the cause. The cause is client dominance. Update norms differ by on average, a few clients carry nearly all the energy of the summed update, and scaling a single client reproduces the collapse. A controlled training study further shows that which LoRA factor looks shared across clients depends on the initialization protocol, so a fix should act on the product . We propose \oursfull (\ours), a data-free server-side step for one-shot federated LoRA that shrinks only the clients whose update norm is an outlier of the pool. It bounds dominance but does not remove it. The alternative bounds (clipping at the median as in robust federated learning, shrinking the whole update, and equalizing every client) fail in the ways the diagnosis predicts. All operators we test except task arithmetic and DC-Merge (which flattens itself) also need each client's spectrum to be flattened first (PolCap). With task arithmetic, has the highest retention among the rules that regress on neither pool, and the best worst-case regression against the unmerged base among the rules we measure on every axis at the largest draw.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
one-shot federated learning, LoRA, PEFT, model merging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JnM4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work identifies client dominance as a major issue for one-shot federated LoRA merging. They further explain the A/B-sharing disagreement by showing that cross-client A-similarity is a property of the initialization protocol. Then, they propose Outlier Capping (OCap), a data-free server-side step for one-shot federated LoRA that shrinks only the clients whose update norm is an outlier of the pool.

Critical strengths
  1. This work provides a clear and testable reason for one-shot federated LoRA merge failure. Instead of saying only that one-shot merging fails because LoRA subspaces point in different directions, it measures a 708x difference in client update norms, shows that three clients contain 99.5% of the energy of the summed update, and shows that scaling one client can cause the same collapse.
  2. The authors show a new finding about LoRA factor sharing, i.e., cross-client A-similarity is a property of the initialization protocol.
Critical weaknesses
  1. The writing of this work is not good; it's hard to understand this work despite careful reading.
  2. The motivation for the one-shot federated LoRA merging is not clear. Why do we need one-shot federated LoRA merging? What are the benefits compared to iterative learning?
  3. The proposed cap rule is based on a hand-set choice. The median-plus-three-MAD rule is fixed before the tests, but the result may change with the number of clients, the number of outliers, or whether the norm is computed per matrix, layer, or full adapter. Tests over these choices and over different levels of client dominance would make it clearer when the method works.
  4. The best result uses more than one change. The best result is OCap with task arithmetic, while many other merge methods also need PolCap. Thus, It is hard to tell how much of the gain comes from OCap itself. Users also do not yet get a simple rule for choosing OCap, PolCap, task arithmetic, or a spectrum-based method.
Questions for authors
  1. How sensitive is OCap to the number three in the median-plus-MAD rule when the number of clients or the gap between large and small updates changes?
  2. Have the authors tested OCap on adapter pools with no clear norm outliers, or on synthetic pools where the dominance level changes gradually?
  3. What happens when clients use different LoRA ranks, scaling factors, quantization settings, or target modules?
  4. Can the authors provide a simple rule for choosing plain OCap, PolCap, task arithmetic, or a spectrum-based merge method?
  5. Could the norm statistics reveal information about a client's training method or data size in a real federated system?
Reviewer_CLT9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on federated lora-based one-shot model merging. Motivated by the different norms across client updates, this paper proposes Outlier Capping. The experiments demonstrates its effectiveness.

Critical strengths
  1. The diagnosis is interesting. The paper demonstrates the norm mismatch through a controlled experiments with FinLoRA, which clearly supports their motivations.
  2. The experiments are comprehensive. The proposed method is effective in handling one-shot merging of FedLoRA.
Critical weaknesses
  1. The problem or setting itself is not clearly stated. What is Federated One-shot Merging? How does it differ from naive data-free model merging? Why it is called 'federated'? This really makes me confused.
  2. The logic of paper writing is problematic. The diagnosis 2 seems useless in both motivation uncovering and method design. It's more like a independent discovery rather than an important part supporting this paper.
  3. The algorithm is more like a MAD-based norm clipping. The main novelty is supported by the uncoverred issue I think.
  4. The format and the writing is problematic in my view. The abstract provided in Openreview systems keeps latex code \ours as the method name, makes us difficult review the paper. Besides, the title is too ambiguous. In Figure2, the subfigures are overlapped.
Questions for authors
  1. How does Federated One-shot Merging differ from naive data-free model merging? Why it is called 'federated'? This really makes me confused.
  2. What is the role of diagnosis 2 in this paper? How does it motivates the method design?
Reviewer_1GNfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies one-shot federated merging of independently trained LoRA adapters. It identifies client-update magnitude imbalance as a potential cause of merge failure, examines the effect of LoRA initialization, and proposes Outlier Capping (OCap) to limit unusually large client updates before merging. The method is evaluated across multiple merge operators and adapter pools.

Critical strengths
  1. The controlled scaling experiment provides direct evidence that client-update magnitude imbalance can cause merge failure without changing update directions.

  2. The paper reports unsuccessful alternatives and negative results, providing useful evidence about the limitations of different magnitude-control strategies.

Critical weaknesses
  1. The algorithmic novelty is limited, and the causal interpretation is too broad. Section 2 states that OCap uses the same norm-clipping mechanism as prior robust federated learning methods (e.g., Nguyen et al., 2022), but places the threshold at the outlier boundary. The dominance experiment shows that magnitude imbalance can cause failure, not that it is the only cause: several operators still require spectral flattening, and a preregistered prediction about this step fails for three of four operators. On LoRA Land, OCap+TA exceeds median clipping+TA by only 0.007, while full equalization+TA performs 0.018 better.

  2. The final method emerges from an exploratory search over 114 rules, while the reported uncertainty estimates do not appear to account for selecting the winner. Baseline tuning is also not fully comparable: task arithmetic uses the peak of a lambda sweep, Iso-C/Iso-CTS are evaluated without their original validation-based multiplier tuning, and DO-Merging uses author-chosen constants. These issues weaken the evidence for the final configuration's general advantage.

Questions for authors

Can the authors establish whether OCap with task arithmetic outperforms median clipping and full equalization on a dominated adapter pool not used during rule selection? The final configuration was selected after exploring 114 rules, and the LoRA Land results do not show a consistent advantage over these simpler alternatives.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.