V-HFLoRA: Target-Basis Aggregation for Heterogeneous Federated LoRA

Vision foundation models (VFMs) can be efficiently adapted with LoRA, but existing federated LoRA methods typically assume a shared backbone or compatible adapter parameterizations. With heterogeneous VFMs, differences in layer correspondence, dimensionality, and backbone-induced parameter geometry can make direct cross-model aggregation geometrically inconsistent or undefined. We propose V-HFLoRA, which represents LoRA updates in coordinates induced by their frozen backbones and identifies corresponding layers through functional similarity. Matched source updates are then transported into the target geometry and combined through support-aware masking and reliability-weighted aggregation before low-rank recovery. We establish target-side convergence by separating the population discrepancy of the ideal collaborative direction from approximation errors due to functional matching, compatible transport, aggregation, and rank-constrained recovery. Across heterogeneous-backbone benchmarks, V-HFLoRA consistently outperforms local training, same-backbone aggregation, aligned averaging, dimension-invariant heterogeneous heterogeneous LoRA methods, and prediction-space distillation. Ablation studies further isolate the contributions of update-conditioned geometry, functional matching, compatible transport, and target-space aggregation.