- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BaryLoRA, which studies how multiple clients can jointly fine-tune a large model while transmitting only small LoRA adapters and more effectively combining their training results. BaryLoRA aggregates the effective update matrices. It uses layer-wise activity summaries derived from optimizer states retained across communication rounds on each client to adjust the weights based on sample counts. Clients perform adaptive updates in the product space and maintain a low-rank representation through rank-constrained projection. The server forms a weighted mean of the effective updates using the adjusted weights, then applies a QR-core decomposition to compute its optimal low-rank approximation and produce a shared adapter. Communication consists mainly of low-rank factors and one state summary per layer; the full optimizer states remain on the clients.
Critical strengths
The method aggregates effective updates directly, avoiding the cross-client terms that arise when the two LoRA factors are averaged separately. Propositions 1 and 2 and Appendix A clearly explain the rank-constrained solution, its non-uniqueness when singular values are tied, and the equivalence between QR-core decomposition and dense projection. The numerical audit in Table 5 further supports the aggregation invariant under fixed weights. Together, these results make the method’s mathematical meaning and implementation objective relatively clear.
Critical weaknesses
1、BaryLoRA adjusts layer-wise aggregation weights using summaries of the second moment of each client’s historical gradients. Greater gradient activity, however, does not necessarily mean that an update is more valuable; it may also reflect data noise, task difficulty, or client drift. The paper explicitly treats this mapping as an empirical design choice. Its theoretical results establish the uniqueness of the weights given the state-based adjustment factors and the properties of aggregation under fixed weights, but they do not explain why the mapping should improve task performance.
2、BaryLoRA introduces several hyperparameters, including the strength of state-based weighting, the range used to clip the state summaries, and the local-prior coefficient. These directly affect how client contributions are weighted and how local optimization proceeds. The paper does not adequately explain how these values were chosen or whether the gains persist across a broad parameter range. In particular, removing the local prior causes substantial performance drops on some tasks, making the method’s dependence on prior strength worth examining. Without this evidence, it is difficult to judge whether the method requires careful tuning for each task or whether its gains will transfer reliably to new data distributions and federated settings.
Questions for authors
1、Figure 1 is described as an overview of BaryLoRA, but it mainly illustrates the limitations of factor averaging, the product-space representation, and weighted barycentric aggregation. It does not adequately show the client-side optimization procedure that is distinctive to this work. In particular, the figure omits the recovery of the update direction in product space, the first- and second-moment states retained across communication rounds, the rank-constrained projection after local updates, and how the state summaries are converted into layer-wise aggregation weights.
2、Although BaryLoRA keeps communication costs low by transmitting low-rank factors and a small number of state summaries, clients must still maintain a dense effective update matrix and first- and second-moment optimizer states, then perform a low-rank projection after local updates. These operations increase local memory and computation costs, potentially limiting the method’s applicability on resource-constrained devices.