- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work identifies client dominance as a major issue for one-shot federated LoRA merging. They further explain the A/B-sharing disagreement by showing that cross-client A-similarity is a property of the initialization protocol. Then, they propose Outlier Capping (OCap), a data-free server-side step for one-shot federated LoRA that shrinks only the clients whose update norm is an outlier of the pool.
Critical strengths
- This work provides a clear and testable reason for one-shot federated LoRA merge failure. Instead of saying only that one-shot merging fails because LoRA subspaces point in different directions, it measures a 708x difference in client update norms, shows that three clients contain 99.5% of the energy of the summed update, and shows that scaling one client can cause the same collapse.
- The authors show a new finding about LoRA factor sharing, i.e., cross-client A-similarity is a property of the initialization protocol.
Critical weaknesses
- The writing of this work is not good; it's hard to understand this work despite careful reading.
- The motivation for the one-shot federated LoRA merging is not clear. Why do we need one-shot federated LoRA merging? What are the benefits compared to iterative learning?
- The proposed cap rule is based on a hand-set choice. The median-plus-three-MAD rule is fixed before the tests, but the result may change with the number of clients, the number of outliers, or whether the norm is computed per matrix, layer, or full adapter. Tests over these choices and over different levels of client dominance would make it clearer when the method works.
- The best result uses more than one change. The best result is OCap with task arithmetic, while many other merge methods also need PolCap. Thus, It is hard to tell how much of the gain comes from OCap itself. Users also do not yet get a simple rule for choosing OCap, PolCap, task arithmetic, or a spectrum-based method.
Questions for authors
- How sensitive is OCap to the number three in the median-plus-MAD rule when the number of clients or the gap between large and small updates changes?
- Have the authors tested OCap on adapter pools with no clear norm outliers, or on synthetic pools where the dominance level changes gradually?
- What happens when clients use different LoRA ranks, scaling factors, quantization settings, or target modules?
- Can the authors provide a simple rule for choosing plain OCap, PolCap, task arithmetic, or a spectrum-based merge method?
- Could the norm statistics reveal information about a client's training method or data size in a real federated system?