- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work the authors propose BiFA, an approach for transferring task vectors across pretrained models with different widths, depths and pretrainings. The authors observe that layer-wise parameter updates have a bilinear structure involving input activations and output gradients, thus they propose to align these two factor spaces across source and target models. The source task vector is then transformed through input-side and output-side mappings estimated from a calibration set. The method is evaluated on vision and language benchmarks.
Critical strengths
- The bilinear interpretation provides a clear and intuitive motivation for separately aligning activation and gradient spaces.
- The manuscript is clear and well written and the method is relatively simple despite addressing a difficult setting.
- The empirical evaluation is extensive, covering different width and depth changes and different architectures, with strong results.
Critical weaknesses
-
Relationship to Theseus. The task vector transport follows a two-sided linear transformation, which is largely inspired by Theseus. The main methodological distinction lies in how the two correspondence maps are constructed, particularly the use of task-conditioned output gradients. The resulting formulation is meaningful and empirical effective, but the degree of methodological novelty over Theseus is limited.
-
Depth/Layer matching is not sufficently clear. For different-depth models the paper states that each target layer is associated with a source layer through relative-depth matching and that the corresponding source task vector is transferred. However it is unclear how this is implemented in practice and how does it behave when several target layers corresnpond to the same source layer (e.g., ViT-L to ViT-B). In particular, does the method directly use the activations/gradients and task vector of the selected source layer to construct each target-layer, or is an intermediate depth-matched source task vector/model explicitly materialized? This may effect also the comparison with Theseus, since this competitor materializes the expanded source model.
Minor:
- The approach requires labeled calibration examples so the meaning of "training-free"should be stated more precisely.
- The motivation for the use of CCA instead of CKA should be better clarified since one is used as a mapping estimator and the other appears in the analysis of layer correspondence.
Questions for authors
- Could the authors clarify the exact implementation of relative-depth matching? When source and target depths differ, do they directly reuse the matched source layer's activations, gradients, and task vector to construct each target-layer update, or do they first materialize a depth-matched task vector/model? If this differs from the strategy used by Theseus, it would be valuable to compare BiFA and Theseus under both depth-transfer strategies, to disentangle the contribution of factor alignment from the depth-matching procedure.
4 comments
[W3] Clarification of Training-Free Setting
Answer: We thank the reviewer for raising this point. We clarify that “training-free” refers to the absence of parameter updates or iterative optimization during task-vector transfer, rather than the absence of labeled data or gradient computation. BiFA uses a small labeled calibration set to perform a single forward–backward pass at the pre-trained initialization, extracting activations and gradients to estimate the alignment maps. We have clarified the definition in the revised manuscript (L70-71).
[W4] Clarification on CKA vs. CCA
Answer: We thank the reviewer for highlighting this point. CKA [G11] and CCA [G12] serve distinct purposes in our study. CKA summarizes representational similarity into a single scalar score, allowing us to assess whether different layers exhibit similar representation structures. However, this score does not specify how individual feature directions correspond across the two spaces, and therefore cannot directly define an alignment map. In contrast, CCA explicitly identifies pairs of canonical directions that maximize correlation between the two representation spaces. These paired directions provide coordinate correspondences from which a linear alignment map can be constructed. Accordingly, we use CKA to analyze layer-wise representational similarity, while employing CCA to estimate explicit cross-model mappings. We have clarified the distinct roles of CKA and CCA in Appendix E (L1349-1367) of the revised manuscript.
Thank you to the authors for clarifying the raised concerns. I will keep my positive score.
Regarding the performance of BiFA on different depth models, looks like that most of the gain is given by the depth matching strategy. Similar gain are not observed on Theseus. Do the authors have some qunatitative insights on the improvement of BiFA, for example the actual alignment values?
We thank the reviewer for the positive assessment and insightful follow-up.
To quantitatively investigate why depth-matching strategies affect BiFA and THESEUS differently, we evaluate functional alignment under both strategies for ViT-B/16 → ViT-L/14. Specifically, we measure the cosine similarity between feature changes induced by transferred task vectors and those induced by target-model fine-tuning, both relative to the same pretrained target model. This metric evaluates how closely the transferred updates reproduce the target's task-specific functional effect, consistent with our objective in Section 3.1.
| Method | Depth Matching | Feature Cosine ↑ | Accuracy (%) |
|---|---|---|---|
| THESEUS | Interpolation (as in THESEUS) | 0.711 | 77.28 |
| BiFA-P | Interpolation (as in THESEUS) | 0.720 | 79.97 |
| THESEUS | Relative depth (as in Ours) | 0.698 | 76.25 |
| BiFA-P | Relative depth (as in Ours) | 0.735 | 84.03 |
BiFA-P achieves higher functional alignment under both depth-matching strategies, consistent with its higher transfer accuracy. Furthermore, relative-depth matching improves both functional alignment and accuracy for BiFA-P, whereas both decrease for THESEUS. This suggests that the effectiveness of depth matching depends on the underlying transfer method.
To further investigate why BiFA-P benefits from relative-depth matching, we extend the output-gradient correspondence analysis from Appendix D.2 (Fig. C) to the depth-mismatched setting. Specifically, we measure the top-10% subspace overlap between the aligned source and target output gradients under both depth-matching strategies.
| Depth Matching (BiFA-P) | Gradient Subspace Overlap ↑ |
|---|---|
| Interpolation (as in THESEUS) | 0.496 |
| Relative depth (as in Ours) | 0.577 |
Relative-depth matching yields higher output-gradient subspace overlap (+16.3%), indicating stronger correspondence between the dominant gradient directions of the matched source and target layers. Together with the functional alignment results, this suggests that BiFA-P benefits particularly from relative-depth matching through improved gradient-space correspondence.
Importantly, BiFA-P already outperforms THESEUS under the same interpolation strategy, indicating that its advantage is not solely attributable to depth matching. We have added these analyses to Appendix G.3 (L1753-1763).
Thank you for the insightful question and discussions that have further solidified the quality of our work.
Dear Reviewer fFZu, We sincerely thank you for your careful review and constructive suggestions. We particularly appreciate your positive assessment of the clear and intuitive bilinear interpretation, the simplicity and clarity of our framework, and the extensive empirical evaluation across different architectures and model configurations. Your comments on the distinction from THESEUS and the depth-matching procedure have helped us further clarify the contributions and implementation of BiFA. We address your concerns and questions in detail below.
Answer:We thank the reviewer for the careful comparison. While BiFA and THESEUS share a two-sided linear transport structure, BiFA's central contribution lies in identifying alignment spaces from the structure of task-vector formation, rather than simply using task-conditioned output gradients.We further examine the structural basis for this choice and provide additional subspace analysis in our response to Reviewer kdux (W1, Theoretical justification of signal choices). As revealed by the bilinear decomposition in Eq. 5, the task vector's column and row spaces are directly related to the spans of output gradients and input activations, respectively. Importantly, since alignment maps are constrained by the subspaces spanned by their calibration signals, effective transfer requires these signals to adequately capture task-vector-relevant directions. Our additional subspace analysis shows that initialization gradient subspaces capture substantially more task-vector energy than output-activation subspaces at matched ranks (33.98% vs. 15.79% at 10% rank).
Consistent with this analysis, Table 5 shows that activation–gradient alignment achieves the highest transfer performance across all three mapping estimators: Procrustes, Ridge, and CCA. Table 6 further supports the importance of task-conditioned gradients. Furthermore, our additional experiments across different source optimizers, larger-scale LLMs, and task-vector addition and negation demonstrate consistent improvements over THESEUS (Appendices B.1 and G.4–G.5). These consistent improvements across diverse settings further support our central motivation that identifying alignment spaces from the bilinear structure of task-vector formation enables effective and generalizable cross-model task-vector transfer.
Answer:We thank the reviewer for pointing out this ambiguity. BiFA does not construct an intermediate depth-matched model. Instead, each target layer j is assigned a source layer using i(j)=round(j(LA−1)/(LB−1)), as described in Section 5.2. The corresponding source task vector is reused, while the activation- and gradient-based alignment maps are estimated independently for each target layer. Consequently, multiple target layers may share the same source task vector but receive different aligned updates.To further examine whether the improvements stem from the alignment method or the depth-matching procedure, we conduct a controlled comparison by applying both interpolation-based and direct layer matching to THESEUS and BiFA-P.
BiFA-P consistently surpasses THESEUS under either matching strategy, with gains of 2.69 and 7.78 percentage points under interpolation and direct matching, respectively. This confirms that the advantage of bilinear factor alignment is not merely a consequence of the depth-matching choice.
Furthermore, the effect of depth matching differs between the two methods. While THESEUS shows only a modest difference between interpolation and relative depth matching (77.28% vs. 76.25%), relative depth matching improves BiFA-P by 4.06 percentage points. We hypothesize that this advantage stems from the interaction between relative depth matching and bilinear factor alignment. Specifically, directly aligning the original source activations and gradients with each target layer may yield more effective target-specific alignment maps than using statistics from interpolated intermediate layers.
We have clarified the layer-matching implementation in Section 5.2 (L405–407) and added the controlled comparison to Appendix G.3 (L1726–1752) of the revised manuscript.