- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In federated LoRA training, the authors reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-rinput-side space, whereas Share-B/Local-A requires a common rank-r output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, they propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, they design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-rinput subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone.
Critical strengths
- The paper is well written and easy to follow.
- The motivation of this work is clear.
- The finding about the difference between Share-A/Local-B and Share-B/Local-A is interesting.
Critical weaknesses
- There are some flaws in the method design. The current method is that, for an LLM, a fixed sharing policy is chosen without considering parameters related to federation, such as the number of clients and the data the client provides. It seems not reasonable.
- The result does not correspond to the description. For example, in Figure 1, the results of Share-B/Local-A are always better than Share-A/Local-B; why do the authors state, "This observation does not imply that Share-B/Local-A is always preferable." (Line 77). Are there any results where Share-A/Local-B is better than Share-B/Local-A?
- In Table 1, the authors' method, FedAS-LoRA, is superior to the previous baseline FedSA-LoRA. However, the authors' method only selects share A or share B. When the method selects share A, isn't it the same as FedSA-LoRA? Will the performance still be better than this baseline in this way? Or is it that the methods chosen by the authors in Table 1 are all share B?
Questions for authors
- How sensitive is the RSS choice to the number of clients, the non-IID settings, and the LoRA rank?
- Would choosing the sharing side separately for each layer or module work better than one global choice?
- How to demonstrate that RSS chooses the right sharing strategy?