acceptodds
Under review as a conference paper at ICLR 2027

V-HFLoRA: Target-Basis Aggregation for Heterogeneous Federated LoRA

Abstract

Vision foundation models (VFMs) can be efficiently adapted with LoRA, but existing federated LoRA methods typically assume a shared backbone or compatible adapter parameterizations. With heterogeneous VFMs, differences in layer correspondence, dimensionality, and backbone-induced parameter geometry can make direct cross-model aggregation geometrically inconsistent or undefined. We propose V-HFLoRA, which represents LoRA updates in coordinates induced by their frozen backbones and identifies corresponding layers through functional similarity. Matched source updates are then transported into the target geometry and combined through support-aware masking and reliability-weighted aggregation before low-rank recovery. We establish target-side convergence by separating the population discrepancy of the ideal collaborative direction from approximation errors due to functional matching, compatible transport, aggregation, and rank-constrained recovery. Across heterogeneous-backbone benchmarks, V-HFLoRA consistently outperforms local training, same-backbone aggregation, aligned averaging, dimension-invariant heterogeneous heterogeneous LoRA methods, and prediction-space distillation. Ablation studies further isolate the contributions of update-conditioned geometry, functional matching, compatible transport, and target-space aggregation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Federated Learning, Multi-modal Learning, Personalization, Distributed Learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_98r2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to aggregate LoRA adapters across federated clients with different frozen vision backbones. The authors break cross-model compatibility into three challenges: layer correspondence, parameter dimensions, and backbone coordinate geometry. They propose V-HFLoRA. The method first selects a subspace dynamically in the singular-vector coordinates of the frozen weights, based on the energy of the current LoRA update. It then uses shared unlabeled calibration data to identify corresponding layers through CCA, ridge alignment, CKA, and monotonic matching. Finally, it transports source updates into the target space, applies a sign mask and reliability weighting in the target coordinates, and recovers client adapters through low-rank approximation.

Critical strengths

Using pretrained backbones with different widths and depths across clients is a realistic scenario in federated learning with heterogeneous resources. Establishing compatibility on the target side during aggregation, without requiring every client to use the same adapter structure, is a promising design direction.

Critical weaknesses

1、The singular directions of frozen weights provide an orthogonal coordinate system within each model, but the paper has not adequately explained why singular directions with corresponding indices in different backbones should have interchangeable functional meanings. In particular, under the cross-dimensional definition in Appendix B.3, replicating coordinates implicitly assumes that the source and target latent axes are already aligned, yet this assumption has not been validated. Moreover, the sign convention of an independent SVD can change the coordinate coefficients, and the paper does not establish whether the transport result is invariant to this arbitrary choice. Thus, the current derivation mainly establishes compatibility in dimensions and coordinate representation; it does not sufficiently support the cross-backbone functional compatibility emphasized by the authors.

2、After simplification, the transport procedure described in the methodological appendix essentially places the source model’s coordinates directly into the target model’s coordinate basis, without establishing a meaningful correspondence between the latent directions of the two models. In contrast, the algorithm box, convergence analysis, and complexity analysis describe a different transport mechanism based on calibration data and orthogonal alignment.

Questions for authors

1、Do the experiments use the basis mapping in Equations (44)–(46) of Appendix B.3, or the calibration-data-based Procrustes alignment in Section C.3.2? Please make the method description, algorithm, and theory consistent, and explain the coordinate-identity simplification and the issue of SVD sign choices noted above.

2、How are the round-wise direction in Equation (72) and the low-rank projection expression in Section C.3.5 derived from the actual adapter aggregation and recovery steps? How do step-size normalization, local training displacement, and unshared coordinates enter the error bound?

Reviewer_wynWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes V-HFLoRA for federated parameter-efficient fine-tuning across heterogeneous vision backbones. The main idea is to aggregate LoRA updates in a target-specific coordinate system derived from the frozen backbone. The method combines update-dependent subspace selection, response-based layer matching, cross-backbone update transport, and conflict-aware aggregation, while preserving each client's native LoRA parameterization. The authors provide a target-side convergence analysis and evaluate the method on three image classification benchmarks, reporting improvements over local training and several federated adaptation baselines. Ablation studies examine the contributions of the proposed components.

Critical strengths
  1. Relevant problem and a practically motivated design objective.

Federated adaptation across frozen backbones with different widths and depths is an important extension beyond heterogeneous LoRA ranks on a shared architecture. Preserving each client's native LoRA parameterization while establishing compatibility at aggregation time is a useful design objective. The proposed target-specific formulation explicitly distinguishes layer correspondence, dimensional mismatch, and coordinate incompatibility, which helps organize the problem.

  1. Useful geometric observation and an attempt at diagnostic evaluation.

The observation that rank⁡(ΔW)≤r\operatorname{rank}(\Delta W)\le r does not imply concentration within the first rr singular directions of the frozen backbone is correct. It provides a reasonable motivation for update-dependent retention. The paper also attempts to separate the effects of matching, retention, masking, and reliability weighting through ablations, rather than relying exclusively on aggregate benchmark accuracy. These could provide useful evidence if the algorithmic and experimental inconsistencies identified below are resolved.

Critical weaknesses
  1. The central cross-dimensional transport is inconsistently specified, and the convergence analysis is not adequately connected to the actual update rule.

In Appendix B.3, Eqs. (44)--(46), let Ui,Vi,Uj,VjU_i,V_i,U_j,V_j denote the retained orthonormal bases with the common dimension qq. The proposed maps are

ΘU=UiUj⊤,ΘV=ViVj⊤.\Theta_U = U_i U_j^{\top}, \qquad \Theta_V = V_i V_j^{\top}.

Substituting these into the transported update gives

ΔW~j→i=(UiUj⊤)(UjCjVj⊤)(VjVi⊤).\widetilde{\Delta W}_{j\to i} = (U_i U_j^{\top})(U_j C_j V_j^{\top})(V_j V_i^{\top}).

Since the retained bases have orthonormal columns, this simplifies to

ΔW~j→i=UiCjVi⊤.\widetilde{\Delta W}_{j\to i} = U_i C_j V_i^{\top}.

Therefore,

Cj→i=Ui⊤ΔW~j→iVi=Cj.C_{j\to i} = U_i^{\top}\widetilde{\Delta W}_{j\to i}V_i = C_j.

Thus, the operation copies the source coefficient matrix into the target basis. The Frobenius-optimality of the ambient maps does not establish that the corresponding source and target singular-coordinate indices have compatible meanings.

This also introduces dependence on arbitrary SVD sign choices. Replacing the source singular vectors by UjD,VjDU_jD,V_jD, where DD is diagonal with entries in −1,1\\{-1,1\\}, leaves the frozen weight unchanged. The same physical source update then has coordinates DCjDDC_jD. Under the stated transport, the target update changes from

UiCjVi⊤U_i C_j V_i^{\top}

to

UiDCjDVi⊤.U_i D C_j D V_i^{\top}.

For q=2q=2, Cj=E12C_j=E_{12}, and D=diag⁡(−1,1)D=\operatorname{diag}(-1,1), the transported update changes sign despite an unchanged source model and source update. No cross-model convention resolving this ambiguity is specified.

However, Appendix C.3.2, p. 29, instead introduces calibration-induced orthogonal Procrustes registrations and analyzes

Cj→i⋆=(RU⋆)⊤CjRV⋆C^{\star}_{j\to i} = (R_U^{\star})^{\top} C_j R_V^{\star}

in Eq. (140). This is generally a different operator. Algorithm 1 and Appendix D also refer to Procrustes registration, whereas Appendix B.2 explicitly states that its response-space ridge map is not used to transport LoRA parameters. The input/output-side responses required for the Procrustes registration are not sufficiently specified in the implementation section. Consequently, it is unclear which transport produced the reported results, and Lemma C.3 cannot directly be treated as a guarantee for Eqs. (44)--(46).

There is a further gap in connecting the optimization analysis to Algorithm 1. The method aggregates absolute adapter coordinates derived from BABA, while Eq. (72) defines the effective round direction as

g^t=−yt+1−ytη\widehat{g}_t = -\frac{y_{t+1}-y_t}{\eta}

Appendix C.3.5 then identifies g^t\widehat{g}_t directly with the recovered aggregate Pr(a^t)P_r(\widehat{a}_t). These quantities are not generally identical. Converting an error in the next adapter parameters into an error in the effective direction introduces a factor of 1/η1/\eta, and hence 1/η21/\eta^2 in the squared error, unless a different update definition is supplied. The abstract inexact-update bound may be valid, but its application to the implemented algorithm has not been established.

These concerns affect the paper's central mechanism and theoretical support, rather than peripheral presentation details.

  1. The experimental evidence cannot currently be interpreted under a consistent and sufficiently reproducible protocol.

First, Table 6 on p. 41 reports λpair=129\lambda_{\mathrm{pair}}=129. In Eq. (33), each selected pair contributes Sab−λpairS_{ab}-\lambda_{\mathrm{pair}}, where the normalized compatibility score lies in [0,1][0,1]. Under the stated partial-matching formulation, if the empty matching is admissible, every nonempty matching has a negative objective and the optimum is empty. Consequently, this configuration would select no cross-client donors. This may be a typographical error, but the actual value and its use in the reported experiments are essential to reproducing the method.

Other settings and results also require reconciliation. Table 6 specifies kpair=5k_{\mathrm{pair}}=5, whereas Table 12 highlights kpair=4k_{\mathrm{pair}}=4 with the main accuracy of 76.4376.43; the result for 55 is 76.2776.27. Table 3 reports 71.8171.81 for "Aligned average" and 70.6270.62 for "Compatible transport + average", although Appendix F describes Aligned FedAvg as using the same matching, transport, and recovery procedures. Different averaging weights or other settings could explain this difference, but they are not specified clearly enough to support the claimed attribution to transport.

Finally, the gains over the strongest main baseline, Co-LoRA, are relatively small, but the manuscript does not define the reported ±\pm quantities or provide the number of independent runs and random seeds. It is therefore not possible to assess the robustness of the improvements. Appendix F also lacks a dedicated description of the Co-LoRA implementation and its method-specific settings, while the origin of the calibration data and its separation from evaluation data are not stated.

These omissions do not demonstrate that the experiments are invalid. They do, however, prevent reliable verification of the main comparative and mechanistic conclusions.

Questions for authors
  1. Clarification of the actual transport: Did the experiments use Eqs. (44)--(46), which simplify to Cj→i=CjC_{j\to i}=C_j, or the calibration-based Procrustes operator in Eq. (140)? If the latter, please define the input/output-side calibration responses and the complete registration procedure. If the former, how is the dependence on arbitrary SVD sign choices addressed, and what establishes cross-model coordinate correspondence?

  2. Clarification of the theoretical update: Please write the exact round-level parameter update implemented by Algorithm 1 and derive its relationship to g^t\widehat g_t in Eq. (72). In particular, how does the adapter-aggregation error in Appendix C.3 become the effective-direction error in Theorem C.6, including the step-size scaling and the treatment of coordinates outside the retained support?

  3. Clarification of the experimental configuration: What values of λpair\lambda_{\mathrm{pair}} and kpairk_{\mathrm{pair}} were actually used for the main results? Please reconcile Table 6 with Eq. (33) and Table 12. What precise difference in averaging rules or experimental settings explains the two transport-plus-average results in Table 3?

  4. Clarification of the strongest-baseline comparison: What do the ±\pm values represent, how many independent runs were performed, and were methods evaluated using matched data partitions and seeds? Please provide the Co-LoRA implementation details, method-specific hyperparameters, alignment procedure, and trainable-parameter and communication budgets needed to interpret the comparison.

  5. Clarification of calibration and evaluation data: Where does the calibration set come from, how is it sampled, and is it disjoint from the evaluation data? Please also clarify how the client-specific training and test partitions are constructed.

Reviewer_vjp6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes V-HFLoRA, a target-specific aggregation framework for federated LoRA fine-tuning across heterogeneous vision foundation models. The method addresses incompatibilities in layer correspondence, parameter dimensions, and backbone-induced coordinate geometry. Experiments using heterogeneous vision backbones report improvements over several baselines.

Critical strengths

The paper addresses a relevant and challenging problem in heterogeneous federated LoRA, where differences in backbone architectures and parameter dimensions complicate cross-client aggregation.

Critical weaknesses
  1. The proposed method aims to establish geometric compatibility between heterogeneous backbones. However, Eqs. (5)–(6) only restrict source and target updates to a common retained dimension, without explaining how their respective singular directions are meaningfully aligned.

  2. The proposed functional matching performs CCA-based adjustment, fits a target-oriented ridge mapping, and subsequently evaluates CKA similarity between the aligned representations. However, the alignment and similarity evaluation appear to rely on the same calibration data. Since CCA and ridge regression explicitly optimize representation correspondence, the resulting similarity scores may be optimistic and may not indicate genuine functional equivalence.

  3. The proposed aggregation relies on coordinate-wise sign consistency to filter incompatible source updates. However, matching coordinate signs is a relatively weak criterion for determining whether a transferred update benefits the target model. Two updates may share the same sign but differ substantially in magnitude or optimization direction.

  4. The convergence results show a potentially nonvanishing residual, however, it is unclear whether this residual is sufficiently small to yield a meaningful convergence guarantee.

  5. The improvements are modest and the figures 1-2 are potentially AI-generated.

Questions for authors

See weaknesses.

1 comment

The authors have not provided a rebuttal, and my concerns remain unaddressed. Therefore, I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.