acceptodds
Under review as a conference paper at ICLR 2027

Optimal Transport Layer Alignment for Cross-Architecture Task Vector Re-basin

Abstract

Task vectors encode task-specific adaptations, but transferring them across architectures is difficult when source and target models differ in width and depth. Existing re-basin methods address width mismatches through coordinate alignment, yet handle depth mismatches using fixed layer correspondences. We formulate task vector re-basin using correspondence weights to combine source representations for alignment and task vector blocks for transport. To estimate these weights, we introduce Optimal Transport Layer Alignment (OTLA), a training-free method that learns adaptive, weighted correspondences between layers of heterogeneous models. Using a small unlabeled calibration set, OTLA combines representational similarity measured by Centered Kernel Alignment with a relative-depth prior in a relaxed optimal transport formulation. The resulting transport plan jointly determines which source layers contribute to each target layer. The weighted formulation supports expansion and compression without modifying the source architecture or fine-tuning the target model. Across eight vision tasks, three Vision Transformer scales, and both model expansion and compression, OTLA achieves the highest average accuracy when integrated into existing re-basin methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Re-basin, Heterogeneous Architectures, Task Vectors, Model Merging, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yFrFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OTLA improves layer correspondence for transferring task vectors between models with different depths and widths. A relaxed optimal transport problem combines activation similarity and relative depth to obtain soft layer assignments. The same weights combine source activations and task-vector blocks before coordinate alignment. The method uses calibration data and validation-based scaling, without optimizing target-model parameters.

Critical strengths

The contribution is supported by comparisons with reasonable layer-matching alternatives, including Nearest Depth and Kernel matching, and by ablations of the transport design. With BiCo, replacing Nearest Depth with OTLA improves average accuracy in all four transfer directions in Table 1: 87.39→88.59, 87.47→88.35, 81.74→83.84, and 80.42→82.98. The larger gains for deeper-to-shallower transfer make this a useful extension of the alignment pipeline.

Critical weaknesses

The formulation in Eq. (3) is inconsistent with width reduction. The activation dimensions imply R∈RdS×dTR \in \mathbb{R}^{d_S \times d_T}, whereas the stated constraint RR⊤=IdSRR^\top = I_{d_S} cannot hold when dS>dTd_S > d_T. Algorithm 1 refers to the same equation without specifying a different constraint for this case, leaving the mathematical description of the compression experiments incomplete.

The motivation in Section 3 does not accurately describe BiCo's handling of depth mismatch. It associates BiCo with Block Extension and then describes the strategy as supporting only shallow-to-deep transfer. BiCo already uses nearest-depth layer-index matching and reports deeper-to-shallower transfer in its Section 5.3 and Appendix F.3. The discussion therefore overstates the limitations of the existing approach.

Questions for authors

What constraint and solver are actually used when the source width exceeds the target width, and how does that formulation relate to Equation (3)? Please also clarify whether the Nearest Depth baseline corresponds to the original BiCo layer-matching rule.

2 comments

We appreciate the reviewer’s recognition of the comprehensive comparisons with reasonable layer-matching alternatives, the supporting transport design ablations, and the consistent improvements across all transfer directions, particularly for deeper-to-shallower transfer. We address the reviewer’s concerns below.


The formulation in Eq. (3) is inconsistent with width reduction. The activation dimensions imply , whereas the stated constraint cannot hold when . Algorithm 1 refers to the same equation without specifying a different constraint for this case, leaving the mathematical description of the compression experiments incomplete. What constraint and solver are actually used when the source width exceeds the target width, and how does that formulation relate to Equation (3)?

We thank the reviewer for pointing out this inconsistency. The reviewer is correct that the constraint in Eq. (3) applies only when dT≥dSd_T \geq d_S, that is the target width is greater than or equal to the source width. For width reduction (dT<dSd_T < d_S), the correct formulation is:

Ri→j⋆=arg⁡min⁡R:R⊤R=IdT∣∣HˉT,j⋆R⊤−HˉS,i⋆∣∣F2R^{\star}_{i \to j} = \underset{R:R^\top R= I_{d_{T}}}{\arg\min} || \bar{H}^{\star}_{T,j} R^{\top} - \bar{H}^{\star}_{S,i} ||_F^2

Both formulations (expansion in Eq.(3) and compression as defined above) are solved using the Singular Value Decomposition (SVD) of the cross-covariance matrix (HˉS,i⋆)⊤HˉT,j⋆=U⋆Σ⋆(V⋆)⊤(\bar{H}^{\star}_{S,i})^{\top}\bar{H}^{\star}_{T,j}=U^{\star}\Sigma^{\star} (V^{\star})^\top, yielding the closed-form solution is Ri→j⋆=U⋆(V⋆)⊤R^\star_{i\rightarrow j}=U^{\star}(V^{\star})^\top. The compression formulation above is already used in our implementation, so the reported results are unaffected. We will include the correct equation in the revised manuscript and clarify Algorithm 1 accordingly.


The motivation in Section 3 does not accurately describe BiCo's handling of depth mismatch. It associates BiCo with Block Extension and then describes the strategy as supporting only shallow-to-deep transfer. BiCo already uses nearest-depth layer-index matching and reports deeper-to-shallower transfer in its Section 5.3 and Appendix F.3. The discussion therefore overstates the limitations of the existing approach. Please also clarify whether the Nearest Depth baseline corresponds to the original BiCo layer-matching rule.

We thank the reviewer bringing this inaccuracy to our attention. Our description was based on the BiCo implementation provided in the merge-and-rebase repository [C], which employs Block Extension for layer matching, as the original BiCo code is not publicly available. As the reviewer correctly points out, BiCo also supports deeper-to-shallower transfer in Appendix F.3. Moreover, the layer-matching rule described in Section 5.3 of the BiCo paper is mathematically equivalent to our Nearest Depth baseline, up to tie-breaking conventions. We will revise the discussion to avoid overstating the limitations of existing approaches and clarify the relationship between our Nearest Depth baseline and the original BiCo matching rule.

[C] Merge-and-Rebase: A Unified Framework and Evaluation Benchmark for Fine-Tuning, Model Merging, and Rebasin, 2026, Panariello et al.

Thank you for the clarification and the revised manuscript. The revision now specifies the constraints and solution procedures for both expansion and compression. The description of BiCo’s capabilities and its relationship to Nearest Depth matching has also been corrected. These changes address the two concerns I originally raised, and I have no further questions.

Reviewer_8GYQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Optimal Transport Layer Alignment (OTLA) for transferring task vectors between models with different depths and widths. OTLA combines representational similarity and relative depth in a relaxed optimal transport formulation to estimate soft layer correspondences. The resulting weights jointly combine source activations and task-vector blocks for Procrustes-based transport. Experiments integrating OTLA into Theseus and BiCo report improvements across eight vision tasks in both expansion and compression settings.

Critical strengths
  • The paper addresses a relevant limitation of cross-architecture task-vector transfer. Separating layer correspondence from coordinate alignment provides a clear and reasonable design.
  • The experiments cover both transfer directions and two re-basin frameworks. Component ablations provide support for combining representational similarity, depth information, and joint assignment.
Critical weaknesses
  • Unverified coordinate compatibility across source layers. OTLA aggregates source activations and task-vector blocks before coordinate alignment. However, matching computational roles does not establish a shared feature basis across depths, and high CKA similarity cannot ensure this compatibility. Further experiments are needed to verify whether aggregation preserves useful information or introduces cancellation.

  • Unclear connection between alignment quality and transfer utility. The reported CKA–Procrustes correlations link representation similarity to alignment quality, but do not establish whether either predicts beneficial task-update transfer. Direct validation against downstream performance gains would strengthen the method’s justification.

Questions for authors
  • How does averaging before alignment compare with aligning each source layer first and then aggregating the transported updates, using the same layer weights?
  • Do higher CKA similarity or lower alignment residuals predict greater target-task improvements from the transported updates across computational roles and transfer directions?
2 comments

We thank the reviewer for their constructive feedback and for recognizing the relevance of cross-architecture task-vector transfer, the clear design separating layer correspondence from coordinate alignment, and the value of evaluating both transfer directions across two re-basin frameworks with supporting component ablations. We address the reviewer’s concerns below.


Unverified coordinate compatibility across source layers. OTLA aggregates source activations and task-vector blocks before coordinate alignment. However, matching computational roles does not establish a shared feature basis across depths, and high CKA similarity cannot ensure this compatibility. Further experiments are needed to verify whether aggregation preserves useful information or introduces cancellation. How does averaging before alignment compare with aligning each source layer first and then aggregating the transported updates, using the same layer weights?

We thank the reviewer for this insightful observation. We agree that aggregating before alignment may introduce cancellation due to incompatible features. To test this, we compared OTLA, under Theseus re-basin, with a variant that aligns each source layer first and then aggregates the transported updates, using the same layer weights, as suggested by the reviewer:

SettingMethodCarsDTDEuroSATGTSRBMNISTRESISC45SVHNSUN397Avg
B → LTarget zero-shot93.0266.3772.5158.5586.6472.5467.7373.5573.86
Align then aggregate92.9873.3588.9378.5593.5174.6285.4874.1982.70 (+8.83)
Aggregate then align (OTLA)92.9971.6389.1477.2596.0776.0785.0873.9882.78 (+8.92)
L → BTarget ZS88.8857.4555.8855.2176.4869.1962.5870.0166.96
Align then aggregate88.9364.4867.7862.7188.9472.8477.4070.5874.21 (+7.25)
Aggregate then align (OTLA)88.9960.3476.8362.9387.9771.9279.6570.4174.88 (+7.92)

Aligning before aggregating achieves comparable average accuracy to OTLA but is more computational expensive, as it requires separate Procrustes alignments for each selected source-target pair, whereas OTLA computes a single alignment per target layer. Both approaches use the same transport weights. We will include these experiments and discuss alternative aggregation strategies as a potential direction for future work in the revision.


Do higher CKA similarity or lower alignment residuals predict greater target-task improvements from the transported updates across computational roles and transfer directions?

We thank the reivewer for the question. While the CKA-Procrustes relationship is already analyzed in the paper (Figure 2 and Appendix F.6), here we further investigate whether lower alignment residuals are associated with better target-task accuracy. For each layer matching strategy, we compute the average Procrustes error as the squared alignment residual divided by the number of observations and the reconstructed feature dimension. We then average the error across layers, roles and eight downstream tasks, and compare it with the average test accuracy.

Using Theseus re-basin, we evaluate both tranfer directions within the same pre-training setting (DataComp) and expansion across different pre-training dataset (LAION -> Datacomp):

Same pretraining (DataComp → DataComp)

SettingMetricBENDKernelOTLA
B → LAvg Procrustes Error ↓0.810.630.590.47
Test Accuracy (%) ↑75.1680.8880.6982.78
L → BNormalized Procrustes Error ↓-0.700.650.62
Test Accuracy (%) ↑-72.4172.9774.88

Different pretraining (LAION → Datacomp)

SettingMetricBlock ExtensionRandomNearest DepthKernelOTLA
B → LAvg Procrustes Error ↓1.941.631.491.271.05
Test Accuracy (%) ↑74.2974.9777.9377.1081.44

Trasfer across different pre-training exhibits higher Procrustes errors than within pre-training transfer. Across all settings, OTLA achieves the lowest Procrustes error and highest accuracy, suggesting that better alignment quality is associated with more effective task vector transfer.

Thanks for the response. I have no further questions.

Reviewer_HwJYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OTLA, a many-to-many optimal transport alignment for moving task vectors between transformer architectures with different depths and widths. OTLA combines representation similarity with relative layer-depth costs and relaxes transport marginals to align layers jointly instead of using independent matches. It is integrated into Theseus and BiCo re-basin settings and evaluated for model expansion and compression primarily within OpenCLIP ViT families. This work concerns unequal-depth but otherwise compatible transformer families, not alignment of arbitrary neural networks. Inference uses representations collected from calibration examples and potentially a target-selected merge coefficient. It is important to distinguish absence of gradient-based training from absence of labeled calibration or other task-specific information.

Critical strengths
  1. Joint layer transport is well motivated for unequal-depth models, where one-to-one correspondence can be structurally inappropriate. The design separates representation evidence, depth priors and marginal relaxation into testable components. A many-to-many transport plan explicitly expresses the possibility that one source computation corresponds to several target stages. Joint coupling can also avoid locally attractive but globally inconsistent layer pairings. This is conceptually better matched to expansion/compression than an independent nearest-depth rule.

  2. The strongest ablation holds the similarity cost fixed: on ViT-B/16 to ViT-L/14 Theseus, independent assignment averages 76.40% whereas joint OTLA reaches 82.78% (Table 3). This supports the value of jointly optimizing layer matches, rather than crediting every gain to a different matching cost. Holding the similarity cost fixed in Table 3 is a key identification step. The 76.40-to-82.78 improvement isolates joint transport more cleanly than merely comparing to a weak index mapping. Since representations are otherwise measured the same way, the gap supports the actual algorithmic contribution rather than an unrelated hyperparameter or feature improvement.

  3. The evaluation covers both expansion and compression, different target tasks and limited calibration budgets. Table 4 also reveals where relaxed transport matters most (B-to-L 81.19 to 82.78; L-to-B 74.49 to 74.88), avoiding a claim of uniform benefit. The B-to-L/L-to-B asymmetry in Table 4 identifies where relaxed marginals help most. Such a distinction is useful for practitioners: expansion entails splitting source content among more target layers, whereas compression aggregates different roles. The authors should retain this directional interpretation instead of presenting one pooled improvement.

Critical weaknesses
  1. 'Training-free' is accurate only in the sense of no additional backbone optimization; the method is not calibration-free. Section 5.1 uses unlabeled calibration examples (ten batches of 32 at the primary setting), while target merge weight alpha_T is chosen on a validation set. Clarifying whether target labels are required is important to assessing deployment in label-poor settings. If alpha_T is tuned with target labels, the method remains training-free but has a different application regime from an unsupervised merge. Baseline fairness also depends on whether every comparator receives the same labeled tuning opportunity. A fixed-coefficient control would reveal how much of the final accuracy is due to transport itself rather than favorable post-hoc selection.

  2. Almost all of the evidence is within related pretrained OpenCLIP ViT variants. Success between such models does not yet establish transfer between tokenizers, widely different representation spaces, or language/vision architectures; this is an external-validity limit, not a flaw in the demonstrated comparisons. Shared OpenCLIP initialization makes similarity between hidden representations tractable. Success there does not establish that the transport plan is meaningful across unrelated tokenizers or modality-specific feature spaces. Rather than requiring a large new experiment in rebuttal, a precise compatibility assumption and a within-family failure analysis would make the claim suitably bounded.

  3. Gains depend on direction and comparator. Marginal relaxation adds about 1.59 points for expansion but only 0.39 for compression in Table 4; the paper should foreground comparisons with strong nearest-depth/kernel alternatives rather than primarily weak index-based mappings. Representation extraction and OT solve time should be separated from the merging cost. The smaller compression gain from relaxed marginals is a reminder that cost construction, coupling and relaxation should be evaluated separately. Variation from task and calibration resampling could be comparable with these sub-point differences. The offline activation-extraction and OT solve cost should also be included even though they are not gradient-based training.

Questions for authors
  1. Please describe which data enter (a) the activation/transport similarity estimate and (b) alpha_T selection, noting label availability and whether any test-task information is accessed. Show results with a globally fixed or unsupervised coefficient so the benefit attributable to transport can be separated from target tuning.

  2. Can the authors report transport-plan stability or entropy across a few calibration resamples and relate it to final accuracy for B-to-L and L-to-B separately? This would help explain when relaxed marginals resolve genuine matching ambiguity rather than merely changing averaging weights.

  3. Report wall time, peak memory and scaling with layer count for activation extraction, similarity, OT solving and merge construction. How many task vectors can reuse an alignment plan before its cost is amortized?

4 comments
AuthorsOpenReview ↗

Response to Reviewer HwJY (1/3)

We thank the reviewer for their constructive feedback, and for recognizing the motivation and benefits of joint many-to-many layer transport and the value of evaluating both expansion and compression settings. We address reviewer concerns below.


'Training-free' is accurate only in the sense of no additional backbone optimization; the method is not calibration-free. Section 5.1 uses unlabeled calibration examples (ten batches of 32 at the primary setting), while target merge weight alpha_T is chosen on a validation set. Clarifying whether target labels are required is important to assessing deployment in label-poor settings. If alpha_T is tuned with target labels, the method remains training-free but has a different application regime from an unsupervised merge. Baseline fairness also depends on whether every comparator receives the same labeled tuning opportunity.

The reviewer is correct. Our approach is training-free but not calibration-free. OTLA reuses the calibration set required for Procrustes alignment by the underlying re-basin method (Theseus [A] or BiCo [B]) to also compute the transport cost without using labels. All compared baselines use the same calibration set.

In our experiments, we select the target merge coefficient αT\alpha_T using target validation labels, following the evaluation protocol adopted in Theseus [A] and implemented in the merge-and-rebase repository [C]. To ensure a fair comparison, all reported re-basin methods and layer-matching alternatives (Block Extension, Nearest Depth, Random, and Kernel) use the same target validation set, ensuring the same labeled tuning opportunity. We will better clarify the application regime and the setting in the revised manuscript.

[A] Transporting Task Vectors across Different Architectures without Training (ICML 2026), Rinaldi et al.

[B] Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer (arXiv 2026), Son et al.

[C] Merge-and-Rebase: A Unified Framework and Evaluation Benchmark for Fine-Tuning, Model Merging, and Rebasin, 2026, Panariello et al.


A fixed-coefficient control would reveal how much of the final accuracy is due to transport itself rather than favorable post-hoc selection. Please describe which data enter (a) the activation/transport similarity estimate and (b) alpha_T selection, noting label availability and whether any test-task information is accessed. Show results with a globally fixed or unsupervised coefficient so the benefit attributable to transport can be separated from target tuning.

(a) All layer-matching strategies are evaluated using the same calibration set that the underlying re-basin method (Theseus or BiCo) uses for Procrustes alignment. OTLA reuses this set to estimate the transport plan without using labels.

(b) To evaluate transfer without selecting coefficients on the downstream tasks, we have performed a new experiment using CIFAR-100 as an external development dataset, disjoint from the eight downstream evaluation tasks. We construct a CIFAR-100 task vector and, for each re-basin method, transfer direction, and matching strategy, we select a single coefficient α\alpha using labeled CIFAR-100 validation examples. We then freeze this coefficient and apply it globally unchanged to all eight tested tasks. No downstream task validation or test accuracy is used to select α\alpha:

MethodDirectionTarget zero-shotBERandomKernelNearest DepthOTLA
TheseusB→L73.8674.5973.9978.3677.5979.97
L→B66.96–67.4168.0568.0671.28
BiCoB→L73.8678.2383.6685.7287.2288.58
L→B66.96–76.1479.8881.0782.76

OTLA achieves the highest accuracy in all four settings, exceeding the strongest baseline by 1.36–3.22 percentage points. These results show that its gains persist when the coefficient is selected on a separate dataset and held fixed across downstream tasks, without task specific coefficient selection on downstream tasks. We will add the results of this new experiment using external validation data to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer HwJY (2/3)

Almost all of the evidence is within related pretrained OpenCLIP ViT variants. Success between such models does not yet establish transfer between tokenizers, widely different representation spaces, or language/vision architectures; this is an external-validity limit, not a flaw in the demonstrated comparisons. Shared OpenCLIP initialization makes similarity between hidden representations tractable. Success there does not establish that the transport plan is meaningful across unrelated tokenizers or modality-specific feature spaces. Rather than requiring a large new experiment in rebuttal, a precise compatibility assumption and a within-family failure analysis would make the claim suitably bounded.

We agree with the reviewer that differences in architecture, modality (vision/language), and representation space can affect the success of OTLA and re-basin methods, and that stating our compatibility assumptions would better bound our claims of generality.

To clarify these boundaries, we have performed two targeted evaluations: transfer between language models and transfer between vision models with different pre-training sources.

Language Models. We evaluated Theseus and BiCo on IFEval [D] through the LM Evaluation Harness [E] in a zero-shot text generation setting, considering Qwen 2.5 in expansion (0.5B →\to 1.5B) and compression (1.5B →\to 0.5B). We compare OTLA with the strongest baseline on the four IFEval metrics:

  • Instruction Strict: Percentage of individual instructions followed exactly.
  • Instruction Loose: Perecentage of individual instructions followed with minor formatting tolerance.
  • Prompt Strict: Percentage of prompts where all instructions are followed exactly.
  • Prompt Loose: Percentage of prompts where all instructions are followed with minor formatting tolerance.
MethodDirectionMatchingInstruction StrictInstruction LoosePrompt StrictPrompt Loose
TheseusExpansionKernel34.2635.1924.8226.24
ExpansionOTLA37.5037.9626.2426.95
CompressionRandom27.7828.7020.5721.99
CompressionOTLA32.8734.2623.4025.53
BiCoExpansionNearest Depth39.3541.2029.0831.91
ExpansionOTLA40.2841.6731.9134.04
CompressionNearest Depth29.1731.4819.1521.28
CompressionOTLA31.0235.1921.2825.53

Interestingly, in compression with Theseus, Random Assignment is strongest baseline suggesting a failure of the other matching strategies in this setting. OTLA outperforms the strongest evaluated layer-matching alternatives in both directions under both methods. We will include full results and setting details from these new transfer experiments experimental for all baselines in the revised manuscript.

Different pre-training sources. We also evaluated transfer from a fine-tuned B/16 pretrained on LAION (laion2b_s34b_b88k) to an L/14 pretrained on DataComp (datacomp_xl_s13b_b90k). To asses representation compatibility across different pre-training sources, we report the Procrustes alignment error alongside downstream accuracy:

MethodMetricBlock ExtensionRandomNearest DepthKernelOTLA
TheseusAVG Test Accuracy (%)74.2974.9777.9377.1081.44
AVG Procrustes Error1.941.631.491.271.05
BiCoAVG Test Accuracy (%)74.9983.6286.7085.9487.42
AVG Procrustes Error1.821.470.680.660.57

For each layer matching strategy, we compute the average Procrustes error, as the squared alignment residual divided by the number of observations and the reconstructed feature dimension. We average the error across layers, roles and eight downstream tasks, and compare it with the average test accuracy.

Despite the different representations arrising from differing pre-training distributions, all layer-matching strategies outperform the target zero-shot baseline (73.86%), with OTLA consistently achieving the highest accuracy and lowest Procrustes error. We will include the full results of these new experiments in the revised manuscript.

Compatibility Assumption. These results show that the success of OTLA is not restricted to vision or to models sharing a pre-training source. OTLA and the other layer-matching strategies inherit the underlying re-basin methods’ assumption that source and target features can be approximately aligned through a Procrustes mapping. Our analysis above, averaged on the 8 vision tasks, shows that OTLA reduces the Procrustes residual, suggesting that its layer matching better satisfies this alignment assumption.

We will make this requirement explicit and bound our generality claims to the evaluated settings in the revision.

[D] Instruction-Following Evaluation for Large Language Models (arXiv 2023), Zhou et al.

[E] The Language Model Evaluation Harness (Zenodo 2024), Gao et al.

AuthorsOpenReview ↗

Response to Reviewer HwJY (3/3)

Gains depend on direction and comparator. Marginal relaxation adds about 1.59 points for expansion but only 0.39 for compression in Table 4. Variation from task and calibration resampling could be comparable with these sub-point differences. Can the authors report transport-plan stability or entropy across a few calibration resamples and relate it to final accuracy for B-to-L and L-to-B separately? This would help explain when relaxed marginals resolve genuine matching ambiguity rather than merely changing averaging weights.

We compare balanced and relaxed OT under Theseus re-basin using the same three calibration resamples for both B→L and L→B. We measure normalized entropy (trasport weight dispersion) and argmax agreement (% of target layers with identical highest-weight source layers across resamples). Metrics are computed per plan, averaged over roles then tasks, and reported as mean ± STD across resamples.

SolverNormalized EntropyArgmax Agreement (%)AVG Acc.(%)
B→LBalanced0.0200 ± 0.00013498.57 ± 0.1881.24 ± 0.86
Relaxed0.0282 ± 0.00011497.87 ± 0.0582.30 ± 0.88
L→BBalanced0.0233 ± 0.00034198.50 ± 0.5074.21 ± 0.27
Relaxed0.0860 ± 0.00031396.59 ± 0.1074.72 ± 0.65

Before cumulative selection, we compute normalized entropy Hj=−1log⁡∣Sr∣∑i∈Srwijlog⁡wijH_j=-\frac{1}{\log|S_r|}\sum_{i \in S_r} w_{ij}\log w_{ij} for each target layer jj using the correspondence weights wijw_{ij} in Eq. (13). We then average over target layers. This measures assignment concentration independently of transported mass: 0 means single-source assignment, 1 uniform weights.

Relaxed marginals increase entropy in both directions, allowing correspondence weights to spread across more source layers. Yet, argmax agreement exceeds 96%, indicating stable dominant correspondences across resamples. This flexibility also improve downstream accuracy in both expansion and compression over balanced OT.


Report wall time, peak memory and scaling with layer count for activation extraction, similarity, OT solving and merge construction. How many task vectors can reuse an alignment plan before its cost is amortized?

We measure wall time and peak memory for single-task B→L and B→H transfer, separating activation extraction, similarity computation, OT solving, Procrustes, and merge construction. Our unoptimized implementation builds on merge-and-rebase [C] and runs on one RTX 4090.

For B→L, the runtimes are:

PhaseNearest DepthBlock ExtensionOTLA
Block Extension-5.54 ± 0.47–
Activation Extraction383.47 ± 3.16422.53 ± 33.13383.47 ± 2.15
OTLA CKA + Matching-–57.07 ± 0.39
OTLA Solver-–5.96e-3 ± 1.57e-4
Theseus Procrustes248.71 ± 2.34270.54 ± 0.77274.96 ± 0.71
Merge Construction1.67 ± 0.031.67 ± 0.031.67 ± 0.03
Total (s)633.85 ± 5.26700.28 ± 34.81717.17 ± 2.93
Relative runtime1.00x1.10×1.13×

For B→H:

PhaseNearest DepthBlock ExtensionOTLA
Block Extension-6.57 ± 0.06–
Activation Extraction612.80 ± 2.02618.67 ± 2.03612.80 ± 2.06
OTLA CKA + Matching-–92.38 ± 0.34
OTLA Solver-–6.91e-3 ± 3.8e-5
Theseus Procrustes404.25 ± 1.82440.91 ± 1.22455.82 ± 2.04
Merge Construction2.27 ± 0.032.27 ± 0.032.27 ± 0.03
Total (s)1019.32 ± 1.941068.42 ± 2.671163.27 ± 3.65
Relative runtime1.00×1.05×1.14×

OTLA adds only 2.4% and 8.9% runtime overhead over BE for B→L and B→H, respectively. Nearest Depth (ND) and OTLA share identical activation extraction and merge construction costs, requiring no additional forward passes or task vector transport operations. OTLA's weighted aggregation of source activations adds only 26 and 52 seconds to Procrustes alignment over ND. CKA and layer matching take 57–92 seconds, while OT solver has negligible cost of 6–7 milliseconds.

Despite additional computations, OTLA reduces peak RAM by 4.7–5.2% and VRAM by 5.6–14.3% compared to Block Extension (BE):

Memory (GiB)Nearest DepthBlock ExtensionOTLA
B → LPeak RAM33.64 ± 0.1739.96 ± 0.0338.10 ± 0.15
Peak VRAM4.75 ± 0.006.26 ± 0.005.91 ± 0.00
B→HPeak RAM52.49 ± 0.2160.27 ± 0.3257.11 ± 0.12
Peak VRAM9.13 ± 0.0011.17 ± 0.009.57 ± 0.00

OTLA improves layer matching with limited runtime overhead and lower peak memory than BE and slight overheads wrt to ND, as acknowledged in the limitations. We will include these measurements in the revised manuscript.

[C] Merge-and-Rebase: A Unified Framework and Evaluation Benchmark for Fine-Tuning, Model Merging, and Rebasin, 2026, Panariello et al.


The paper should foreground comparisons with strong nearest-depth/kernel alternatives rather than primarily weak index-based mappings.

We thank the reviewer for the suggestion. We already compare OTLA with the stronger ND and Kernel baselines in the main Tab. 1 and focus on them in Fig. 3. We will further emphasize these comparisons in the revised manuscript.

Thank you for the additional experiments. My main concern was whether OTLA’s gains would persist without tuning the merge coefficient on each downstream task. The new experiment selects the coefficient on a separate CIFAR-100 development set and holds it fixed across the eight downstream tasks. OTLA still leads the strongest comparison by 1.36–3.22 percentage points across the four settings. This addresses the concern that drove my original rating. The calibration resampling analysis and runtime breakdown also make the trade-offs clearer, and the Qwen and cross-pretraining experiments broaden the evidence. I would keep the scope tied to source and target representations that can be approximately aligned. I am raising my rating from 2 to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes

We thank all the reviewers for their constructive feedback. We have uploaded the revised manuscript with all revisions highlighted in blue. In summary, we have:

  • clarified calibration and label requirements, including the shared calibration data and labeled validation sets used by all matching strategies (Section 5.1 and Appendix E.4);
  • added a globally fixed coefficient experiment using CIFAR-100, without downstream coefficient selection (Section 5.2, Table 2, and Appendix E.4);
  • added Qwen2.5 transfer experiments on IFEval in both expansion and compression (Section 5.2, Table 3, and Appendix E.5);
  • added vision transfer experiments across different pre-training sources and clarified the approximate Procrustes alignment assumption (Appendices F and G.9);
  • compared balanced and relaxed OT across calibration resamples, reporting normalized entropy, argmax agreement, and test accuracy (Appendix G.7);
  • compared aggregating before alignment with aligning before aggregation, and analyzed the relationship between Procrustes error and downstream accuracy (Appendices G.8 and G.9);
  • added wall time and peak memory measurements for B→L and B→H transfer (Appendix G.10);
  • added the Procrustes formulation for width reduction and corrected the description of BiCo’s handling of depth mismatch (Section 3, Eq. (3), Section 5.1, and Appendix D).

We again thank the reviewers for their time and engagement with our work. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.