- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how to aggregate LoRA adapters across federated clients with different frozen vision backbones. The authors break cross-model compatibility into three challenges: layer correspondence, parameter dimensions, and backbone coordinate geometry. They propose V-HFLoRA. The method first selects a subspace dynamically in the singular-vector coordinates of the frozen weights, based on the energy of the current LoRA update. It then uses shared unlabeled calibration data to identify corresponding layers through CCA, ridge alignment, CKA, and monotonic matching. Finally, it transports source updates into the target space, applies a sign mask and reliability weighting in the target coordinates, and recovers client adapters through low-rank approximation.
Critical strengths
Using pretrained backbones with different widths and depths across clients is a realistic scenario in federated learning with heterogeneous resources. Establishing compatibility on the target side during aggregation, without requiring every client to use the same adapter structure, is a promising design direction.
Critical weaknesses
1、The singular directions of frozen weights provide an orthogonal coordinate system within each model, but the paper has not adequately explained why singular directions with corresponding indices in different backbones should have interchangeable functional meanings. In particular, under the cross-dimensional definition in Appendix B.3, replicating coordinates implicitly assumes that the source and target latent axes are already aligned, yet this assumption has not been validated. Moreover, the sign convention of an independent SVD can change the coordinate coefficients, and the paper does not establish whether the transport result is invariant to this arbitrary choice. Thus, the current derivation mainly establishes compatibility in dimensions and coordinate representation; it does not sufficiently support the cross-backbone functional compatibility emphasized by the authors.
2、After simplification, the transport procedure described in the methodological appendix essentially places the source model’s coordinates directly into the target model’s coordinate basis, without establishing a meaningful correspondence between the latent directions of the two models. In contrast, the algorithm box, convergence analysis, and complexity analysis describe a different transport mechanism based on calibration data and orthogonal alignment.
Questions for authors
1、Do the experiments use the basis mapping in Equations (44)–(46) of Appendix B.3, or the calibration-data-based Procrustes alignment in Section C.3.2? Please make the method description, algorithm, and theory consistent, and explain the coordinate-identity simplification and the issue of SVD sign choices noted above.
2、How are the round-wise direction in Equation (72) and the low-rank projection expression in Section C.3.5 derived from the actual adapter aggregation and recovery steps? How do step-size normalization, local training displacement, and unshared coordinates enter the error bound?
The authors have not provided a rebuttal, and my concerns remain unaddressed. Therefore, I will maintain my original rating.