acceptodds
Under review as a conference paper at ICLR 2027

From Parameter Rebasin to Functional Transfer Across Heterogeneous Transformers

11 views

Abstract

Adapting pretrained Transformers to downstream tasks produces task-specific updates that are expensive to reproduce for every model variant. Reusing these updates across different depths and widths is challenging because their parameters have incompatible shapes and do not share a canonical coordinate system. We introduce Ariadne, a training-free method that transfers task-specific behavior across heterogeneous Transformers without resizing the source model or transporting its parameters. Rather than treating the source task vector, i.e. the difference in parameter space between the fine-tuned and the base models, as the object to move, Ariadne characterizes fine-tuning by the activation changes it induces at matched blocks. First, it aligns these effects to a target base through local Procrustes maps. Then, it directly synthesizes a native target update by closed-form ridge regression. Across Transformer expansion, reduction and language-model transfer, Ariadne improves target-model performance while substantially reducing the time and memory required to construct an update relative to previous cross-architecture rebasin approaches. These results show that local functional effects provide an efficient alternative to parameter transport.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
rebasin, model transfer, functional transfer, transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aAMpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ARIADNE, a method for transferring the effects of fine-tuning between Transformers with different depths and widths. Using paired calibration inputs, the method collects intermediate activations from the source base model, the source fine-tuned model, and the target base model. It establishes module correspondences based on relative depth, uses local Procrustes mappings to transform the activation displacements induced by source fine-tuning into the target coordinate system, and then fits parameter updates for the final projection layers of the target MLPs via closed-form ridge regression. The resulting updates can be written directly into the target model, without activation interventions at inference time. The default activation-fitting stage requires neither labels nor backpropagation, whereas the gradient variant requires the corresponding supervision.

Critical strengths
  1. Transferring fine-tuning displacements is well motivated: The method preserves the target base representations and transforms the changes induced by source fine-tuning into target parameter updates, avoiding the direct transfer of source parameters with incompatible shapes. The relevant experimental results show that fitting the absolute fine-tuned endpoint performs worse than fitting the fine-tuning displacement, providing empirical support for the choice of transfer target.
  2. The vision experiments demonstrate gains in both accuracy and fitting efficiency: In both vision transfer directions, the default method achieves substantially higher average accuracy than the parameter-transfer baselines included in the comparison. The appendix also reports results under different budgets and across three calibration runs, providing substantial positive evidence from the vision experiments.
Critical weaknesses
  1. Is the closed-form solution for dimensionality-reducing alignment consistent with the stated objective? When the source width exceeds the target width, Equation (1) on page 4 requires the mapping to have orthonormal columns. Equation (2) then directly uses the product of the SVD factors of the source–target cross-matrix as the mapping. This solution maximizes the inner product between the mapped source activations and the target activations. However, the squared-error objective also includes the squared norm of the mapped source activations, which generally varies with the mapping under the orthonormal-column constraint in the dimensionality-reducing case. The authors therefore appear not to have explained the conditions required for Equation (2) to be an optimal solution to Equation (1), or whether these conditions are satisfied in the actual method. Could the authors clarify these points?

  2. Has the accuracy of the fixed-input approximation been validated after assembling the updates? The paper restricts the expression for the change in component outputs (Equation (9)) to a fixed input bank. It further explains that each layer is fitted independently using inputs from the target base model, and that the updates are assembled only after all fitting is complete. However, once upstream layers are updated, the inputs actually received by downstream components will generally differ from those used during fitting. Have the authors considered this effect? Could they report, after all updates have been assembled and using the same evaluation inputs, the error between the actual output displacement of each target module and its corresponding desired displacement, and compare it with the fitting error under fixed inputs? This would quantify the effect of input changes on local displacement matching.

Questions for authors
  1. When the source width exceeds the target width, under what conditions does the SVD solution in Equation (2) minimize Equation (1), and are these conditions satisfied in the actual implementation?

  2. After all updates have been assembled, how much does the error between the actual and desired module output displacements change compared with the fitting error under fixed inputs?

2 comments

We thank the reviewer for the valuable feedback. Below, we address the two mathematical concerns: we correct the stated alignment objective without changing the implementation or reported results, and provide new held-out diagnostics and a sequential-fitting comparison to quantify the effect of assembling independently fitted updates. We will incorporate these clarifications and results into the revised paper.

W1/Q1 Solution to the reduction case.

The reviewer is correct: Eq. (2) does not generally minimize the squared-error objective in Eq. (1) when the source is wider than the target. This is an error in the mathematical description in the paper. Our implementation directly computes the polar factor of the centered source–target cross-covariance, as specified in Eq. (2), in both directions; all reported experiments use this construction.

More precisely, writing X=S~π(j)baseX=\widetilde S_{\pi(j)}^{\mathrm{base}} and Y=T~jbaseY=\widetilde T_j^{\mathrm{base}}, the implemented alignment solves Qj∈arg⁡max⁡R∈Pds,dttr⁡(R⊤X⊤Y).Q_j\in\arg\max_{R\in\mathcal P_{d_{\mathrm{s}},d_{\mathrm{t}}}} \operatorname{tr}(R^\top X^\top Y). For the thin SVD (X⊤Y=UjΣjVj⊤)(X^\top Y=U_j\Sigma_jV_j^\top), a solution is Qj=UjVj⊤Q_j=U_jV_j^\top.

∥XR−Y∥F2=tr⁡(R⊤X⊤XR)+∥Y∥F2−2tr⁡(R⊤X⊤Y).\|XR-Y\|_F^2 = \operatorname{tr}(R^\top X^\top XR) +\|Y\|_F^2 -2\operatorname{tr}(R^\top X^\top Y). When ds≤dtd_{\mathrm{s}}\leq d_{\mathrm{t}}, the constraint RR⊤=IRR^\top=I makes the first term constant, so the two objectives are equivalent. When ds>dtd_{\mathrm{s}}>d_{\mathrm{t}}, the constraint is R⊤R=IR^\top R=I, and the first term generally varies with RR. Isotropic source covariance would suffice for equivalence, but we do not assume it. In reduction, Eq. (2) therefore maximizes aggregate source–target cross-covariance under a semi-orthogonality constraint, but need not minimize squared error. The resulting map does not amplify the source displacement’s norm. We believe this is the right objective for transfer: we want to know where the fine-tuning change should go in the target, not to stretch it to fit the target activations. Consistent with this, maps fitted to minimize reconstruction error transfer worse in reduction: an unconstrained linear map (ridge, Table 2) reaches 76.52%, and a two-layer MLP (our response to Reviewer m37t) 76.33%, versus 79.21% for QjQ_j in the same run.

We will correct Eq. (1) in the main paper to state the cross-covariance maximization objective and explicitly explain this distinction between expansion and reduction. Eq. (2), the implementation, and the reported numerical results remain unchanged.

W2/Q2. Fixed-input approximation after assembly.

The reviewer is right that each regression in Eq. (9) is solved on a fixed input bank HjH_j collected from the target base model. Once all updates are assembled, block jj instead receives inputs already changed by the updates of blocks 1,…,j−11,\dots,j-1. Its realized output displacement can therefore differ from the fixed-input fit. To quantify this gap, as the reviewer suggests, we compare the default independent fitting with a sequential variant that accounts for upstream updates:

Independent fitting (default). Each block is fitted on inputs HjH_j from the target base to reproduce the full displacement DjD_j, as if upstream blocks were unchanged (Eqs. 8–9).

Sequential fitting. Blocks are fitted one at a time, from the first to the last. Each block is fitted on the inputs it actually receives once the upstream ARIADNE updates are applied, and reproduces only what is still missing: Dj−D^jupD_j-\hat D_j^{\mathrm{up}}, where D^jup\hat D_j^{\mathrm{up}} is the displacement after block jj due to the upstream updates alone.

We measure two errors for each target block jj, both normalized by ∥Dj∥F\|D_j\|_F, and evaluated on held-out images disjoint from the calibration set:

  • Local error: how well the block’s own update reproduces its target on the inputs it was fitted on, i.e., ∥HjΔWj+1βj⊤−Dj∥F/∥Dj∥F\|H_j\Delta W_j+\mathbf 1\beta_j^\top-D_j\|_F/\|D_j\|_F for independent fitting and ∥HjΔWj+1βj⊤−(Dj−D^jup)∥F/∥Dj∥F\|H_j\Delta W_j+\mathbf 1\beta_j^\top-(D_j-\hat D_j^{\mathrm{up}})\|_F/\|D_j\|_F for sequential fitting.
  • In-network error: ∥D^j−Dj∥F/∥Dj∥F\|\hat D_j-D_j\|_F/\|D_j\|_F, where D^j\hat D_j is the displacement actually realized after block jj, relative to the target base, when all updates are applied together.

For sequential fitting, the realized displacement is D^jup+HjΔWj+1βj⊤\hat D_j^{\mathrm{up}}+H_j\Delta W_j+\mathbf 1\beta_j^\top, so local and in-network errors coincide by construction; this does not imply zero error with respect to DjD_j.

FittingLocal errorIn-network error
B/16→L/14, block 1both0.290.29
B/16→L/14, median over blocksindependent0.762.48
sequential0.990.99
L/14→B/16, block 1both0.230.23
L/14→B/16, median over blocksindependent0.821.63
sequential0.930.93

Independent updates accumulate. At the first block, no upstream update exists and the two errors coincide. Deeper in the network, independently fitted updates add up: each block reproduces the full displacement on top of what upstream blocks have already produced. With independent fitting, the error grows from 0.76 (local) to 2.48 (in-network) in extension and from 0.82 to 1.63 in reduction.

Why sequential fitting has a higher local error. Sequential fitting removes this accumulation (0.99 and 0.93), but its local error is higher than in the independent case. Independent fitting never has to correct errors coming from upstream blocks: each block only matches DjD_j on the base inputs. Sequential fitting must instead also absorb (i) the residual error of the upstream updates and (ii) the effect of those updates on the non-linear components (normalization, attention, and MLP activations) of the subsequent blocks, a correction that grows deeper in the network. Since an update of the MLP output projection has only partial expressive power, as Adila et al. [1] also show for MLP-output interventions compared with interventions on the full residual stream, each block can satisfy this growing request only partially, which raises its local error.

However, better displacement matching does not uniformly improve accuracy: sequential fitting increases reduction accuracy from 78.96% to 81.06%, but decreases extension accuracy from 84.54% to 82.60%. We therefore retain independent fitting as the default, since sequential fitting does not consistently improve performance and introduces dependencies between block fits.

We will incorporate these diagnostics and the sequential comparison into the revised paper.

[1] Adila et al. Weight Updates as Activation Shifts: A Principled Framework for Steering. ICML, 2026.

Reviewer_7oQrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduce ARIADNE, a training-free method for transferring task knowledge from a model to another. The method estimate an orthogonal map between source base activations and target base activations (for each pair of blocks) and then it uses the source base model and the fine-tuned version to calculate how much activations shifted because of the fine-tuning and then use this source fine-tuning effect to map into the target coordinates and find the desired shift in activations. Finally, this shift is used to compute an additive update to one residual-writing target component in each matched target block.

Critical strengths
  1. The idea is simple and well motivated: the method doesn't need to find parameter correspondences across depths and widht, because it simply transfer the effect of fine-tuning. The method is closed form and the default variant doesn't need labels to update the model.
  2. In the vision setting, ARIADNE is 30-90x faster and uses 2-6x less memory compared to existing methods. While in the LLM experiments it is 7x faster, while achieving better performance.
Critical weaknesses
  1. The setting is quite restricted: the method supposes that both a base model and a fine-tuned version of it exists, as well as calibration data and labeled validation data. Given these, natural baselines or alternatives are not reported:
  • When extending (ViT-B to ViT-L) using directly the fine-tuned version is better than using ARIADNE (i.e., 93.19 vs 84.54 / 86.50). So the need for extension is not clear.
  • fine-tuning the target (or only its heads) on the same labeled samples, or distilling the source fine-tuned model into the target base.
  • for the reduction case motivated in the introduction (same model under a tighter serving budget), applying post-hoc pruning/compression to the source fine-tuned model.
  1. The method is only evaluated to same family architecture (i.e. ViT-B to Vi-L and viceversa for vision, and Qwen for text), and not even all possible configurations are tested. Due to the limited setting, at least more family should be analyzed.
  2. It is true that ARIADNE performs better than existing methods, however improvements are reported with respect to the target base, but accuracy remains far from the fine-tuned models (e.g., 79 vs 95 in vision reduction, 34 vs 76 on GSM8K reduction). Also, on GSM8K, ARIADNE is close to the target base in both directions. The target fine-tuned model (upper bound) should be reported explicitly for each setting to understand how much of the achievable improvement is recovered.

I am willing to raise my score if: (a) a comparison with fine-tuning or distilling the target on the same data (and with pruning in the reduction setting) is added, (b) the targed base fine-tune models are reported and show that ARIADNE recovers a substantial part of the gap, and (c) more model families and larger size gaps are tested (e.g., ViT-Tiny/S and ViT-L, DINO).

Questions for authors
  1. How many samples are used for the transformation estimation and for the ridge fit in the different settings (i.e., B=1, B=10,..) and how are they chosen? How sensitive are results to these choices? Are the results in table 1 already mean over multiple runs for a single dataset? If not, could the authors provide std over calibration runs in the main table for both baselines and ARIADNE in both vision and text domain?
  2. Can the authors clarify how is the task-vector multiplier α\alpha chosen (e.g., which values are tested, how much labeled validation data is needed) and how much time does this search add to the reported construction cost?
  3. The BICO numbers in Table 1 are different from those reported in the paper for the same setting (ViT-B/16 and ViT-L/14, DataComp-XL). Could the authors clarify why?
  4. How does ARIADNE compare with fine-tuning specific components (for example the target heads) on the labeled calibration set? In the reduction setting, how does it compare with training-free pruning [1,2] or few-sample compression [3] or with distilling the source fine-tuned model into the target base on the calibration set, at matched compute?
  5. In Figure 2, could the authors provide CKA between two native fine-tunings of the same task with different seeds and between fine-tunings of two different tasks, to use them as two references value to make the values interpretable?
  6. Do the authors have preliminary results or intuitions on whether the method works across different model families?

[1] Frantar, Elias, and Dan Alistarh. "Sparsegpt: Massive language models can be accurately pruned in one-shot." International conference on machine learning. PMLR, 2023. [2] Sun, Mingjie, et al. "A simple and effective pruning approach for large language models." International Conference on Learning Representations. Vol. 2024. 2024. [3] Zhang, Hanxiao, Yifan Zhou, and Guo-Hua Wang. "Dense vision transformer compression with few samples." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

5 comments

We thank the reviewer for recognizing ARIADNE’s motivation and efficiency. We first address the three conditions for raising the score: (A) adaptation comparisons, (B) native target references, and (C) larger architecture gaps and different pretraining corpora, followed by the remaining questions.

A. Target tuning, distillation, and compression comparisons (condition a; W1/Q4)

A.1. Same-data tuning and distillation

ARIADNE constructs its target update without calibration labels or backpropagation following the rebasin literature. Nevertheless, we compare it with iterative adaptation. All calibration-based methods use the same 320 training images per task (B=10\mathcal B=10); full-data native fine-tuning is a reference. Supervised tuning uses labels; distillation matches source fine-tuned predictions without calibration labels. Both require target backpropagation, unlike ARIADNE. Iterative baselines use validation-selected early stopping. We test head-only, MLP-projection, and full-tower adaptation; MLP-projection updates match ARIADNE’s update locations.

MethodGrad-freeLabel-free constructionExtension acc.Extension TFLOPReduction acc.Reduction TFLOP
Target base✓✓73.87%—66.95%—
ARIADNE✓✓84.54%14078.96%132
Head-only tuning✗✗80.22%5276.72%11
Supervised MLP-proj. tuning✗✗88.26%2,43485.19%418
Supervised full-tower tuning✗✗89.13%2,27486.19%564
Logit distillation, MLP proj.✗✓88.43%2,82085.68%649
Logit distillation, full tower✗✓89.74%3,91887.26%994
Native target full fine-tuning (full-data reference)✗✗94.58%124,00093.19%27,000

As expected, allowing iterative target training generally yields higher accuracy. The practical question is what can be achieved when that additional training is unavailable or undesirable, i.e., the constraint motivating training-free rebasin. ARIADNE offers a strong option here: it exceeds the cheaper head-only baseline by 4.32 points in extension and 2.24 points in reduction, while MLP-projection tuning and distillation require approximately 17–20× its construction FLOPs in extension and 3.2–4.9× in reduction to achieve higher accuracy. Distillation also avoids calibration labels, but still requires iterative target training. ARIADNE’s contribution is therefore to recover useful task behavior at low construction cost without that training requirement.

A.2. Reduction through compression

For reduction, we compare against structured adaptations of SparseGPT and Wanda (reviewer references [1–2]), compressing the fine-tuned ViT-L/14 source to ViT-B/16’s inference-FLOP budget using the same 320 calibration images.

MethodInference GFLOP/imageConstruction TFLOPAccuracy
Source ViT-L/14 fine-tuned (uncompressed)162.0—94.58%
Target ViT-B/16 base35.1—66.95%
ARIADNE→ViT-B/1635.113278.96%
SparseGPT, structured35.122961.21%
Wanda, structured35.11016.02%
Native ViT-B/16 fine-tuning (reference)35.1≈27,00093.19%

Inference GFLOP measures cost per image; construction TFLOP includes calibration passes, statistics, and solves, including passes through progressively pruned networks. Evaluation and selection are excluded.

Matching ViT-B/16 reduces inference computation by ≈78%. Our structured adaptation must remove entire computational groups to achieve this reduction, rather than simply introduce sparsity into individual weight matrices. This is particularly demanding for Wanda: its weight-and-activation importance scores select what to retain, but do not refit surviving weights to compensate for the removed contributions. At this compression level, the resulting functional disruption is consistent with its sharp accuracy collapse.

A.3. Matched-compute comparisons

We additionally evaluate adaptation under fixed construction budgets using identical calibration inputs: 140/132 TFLOP for extension/reduction at B=10\mathcal B=10, and 29/16 TFLOP at B=1\mathcal B=1. Entries below report test accuracy. Iterative methods use the final update at α=1\alpha=1, without coefficient or checkpoint selection; ARIADNE retains validation-selected scaling. Validation and coefficient-selection costs are excluded.

MethodGrad-freeLabel-free constructionB/16→L/14, B=1B/16→L/14, B=10L/14→B/16, B=1L/14→B/16, B=10
ARIADNE✓✓78.3483.5974.2279.63
FT c_proj✗✗77.7782.9573.3084.32
Distill. c_proj✗✓77.7382.6472.8083.69
FT full✗✗77.6384.9573.8285.08
Distill. full✗✓77.4683.8874.0283.78

ARIADNE has the highest numerical accuracy at one batch, with small margins. At ten batches, iterative methods are stronger in reduction and competitive in extension: avoiding target training does not guarantee superior accuracy at matched FLOPs. A.1 uses submitted three-run ARIADNE averages; A.3 uses a separate calibration draw.

B. Native target fine-tuned references (condition b; W3)

Vision. We evaluate native target fine-tuned checkpoints under the same protocol and report mean per-task accuracy normalized by the corresponding target reference:

Normalized accuracy=1008∑i=18Acci(method)Acci(target FT).\mathrm{Normalized\ accuracy} = \frac{100}{8}\sum_{i=1}^{8} \frac{\mathrm{Acc}_i(\mathrm{method})} {\mathrm{Acc}_i(\mathrm{target\ FT})}.

MethodExtensionReduction
Target base73.87% (78.40%)66.95% (72.40%)
ARIADNE84.54% (89.27%)78.96% (84.60%)
Native target fine-tuning reference94.58% (100%)93.19% (100%)

Entries report mean accuracy, with normalized mean per-task accuracy in parentheses. ARIADNE substantially improves on the target base without iterative target training, while leaving a gap to native fine-tuning.

Language. For the Qwen experiments, we additionally evaluate the corresponding native Instruct checkpoints using the submitted prompts, decoding settings, and evaluation items:

Original experimentTarget baseARIADNENative target FT
IFEval, 0.5B→1.5B25.37%43.52%43.83%
IFEval, 1.5B→0.5B19.52%30.56%31.87%

Instruction-following transfer approaches the native target references in both directions. The submitted mathematics gains are modest; without corresponding native math-specialized checkpoints at the target sizes, we cannot quantify the remaining gap for those experiments.

C. Larger architecture gaps (condition c; W2)

Larger vision gap and different pretraining data. We evaluate ViT-B/16 ↔ ViT-H/14, increasing the image-tower parameter gap from 3.5× in the main experiments to 7.3× (86M vs. 632M), with 12 vs. 32 blocks and widths of 768 vs. 1280. We test both LAION-2B-pretrained backbones and transfer between a DataComp-XL-pretrained B/16 and a LAION-2B-pretrained H/14. All settings use Vision8, 320 calibration images per task, and validation-selected scaling.

Pretraining: B/16 / H/14MethodExtension B/16→H/14Reduction H/14→B/16
LAION-2B / LAION-2BTarget base70.97%62.51%
ARIADNE82.07%76.05%
Native target fine-tuning94.75%92.59%
DataComp-XL / LAION-2BTarget base70.97%66.95%
ARIADNE83.34%77.05%
Native target fine-tuning94.75%93.19%

ARIADNE improves accuracy in both directions across this larger architecture gap, including when source and target use different pretraining corpora.

W1. Practical motivation for transfer

Why transfer to a larger model? Cross-architecture reuse is studied by recent works such as Transport and Merge [1], Theseus, and BiCo. It enables adoption of another backbone when original labels or training resources are unavailable. Source-task accuracy alone overlooks the target’s broader capabilities. We evaluated six datasets unused during adaptation: CIFAR-100, STL-10, Flowers-102, Oxford Pets, Food-101, and FER-2013:

ModelAvg. zero-shot accuracy (six unrelated datasets)
Larger target base83.65%
Larger target with ARIADNE82.46%
Smaller source fine-tuned model75.25%

Thus, extension largely retains the target’s broader performance while transferring task-specific behavior. Furthermore, transfer can also exceed the source on the adapted task: submitted Table 3 reports 43.52% vs. 31.87% average strict IFEval accuracy.

[1] Cui et al. Transport and Merge: Cross-Architecture Merging for Large Language Models. ICML, 2026.

Transfer as initialization. When compute is available, rebasin can be complementary to further training: it can reuse source expertise to give target adaptation a head start. We test this by applying the same MLP-projection tuning or logit distillation procedure to either the untouched target base or the ARIADNE-adapted target, using the same 320 images per task and validation-selected early stopping:

AdaptationInitializationExt. accuracyExt. TFLOPRed. accuracyRed. TFLOP
Supervised tuningTarget base88.26%2,43485.19%418
Supervised tuningARIADNE89.56%1,73387.09%486
Logit distillationTarget base88.43%2,82085.68%649
Logit distillationARIADNE90.11%2,93087.30%765

These results show that the ARIADNE initialization improves accuracy in all four comparisons, indicating that the transferred source adaptation remains useful even when the target can learn from the calibration data directly. In supervised extension, it also lowers total construction FLOPs by 29%, including initialization. Thus, transferring existing expertise can improve adaptation beyond what training from the target base achieves.

Q1. Calibration samples and variability

Vision uses 32 images per batch: B∈1,5,10,20,50\mathcal B\in\\{1,5,10,20,50\\} gives 32,160,320,640,1600\\{32,160,320,640,1600\\} images per task. A seeded permutation samples training images without replacement. Alignment and ridge use the same images, with each model’s evaluation preprocessing.

Tables 5–6 report per-task means/std over three calibration runs; Table 1 reports ARIADNE means. At B=10\mathcal B=10, Vision8 accuracy is 84.54±0.19% in extension and 78.96±0.73% in reduction. The new adaptation comparisons use separate calibration draws. Theseus/BiCo averages vary by at most 0.4 points across calibration draws (std: 0.24/0.14), well below their gaps to ARIADNE.

Q2. Multiplier selection and its cost

All three methods select per-task multipliers using the same labeled validation data: a fixed 10% partition of the original test split, with the remaining 90% held out for evaluation. Search scans α\alpha in steps of 0.05, stopping after two consecutive non-improvements.

Across eight tasks at B=10\mathcal B=10, selection takes approximately 18/135/79 minutes for ARIADNE/Theseus/BiCo with a ViT-L/14 target, and 5.5/24/22 minutes with ViT-B/16. Each coefficient has the same evaluation cost across methods—about two minutes across eight tasks for ViT-L/14 and one minute for ViT-B/16—so search cost scales with the number of evaluations. These costs are additional to construction costs in Tables 9–10.

Q3. BiCo reproduction

BiCo results are our reproduction: no code or checkpoints were released, requiring us to reproduce both the method and the fine-tuned models. Matching architectures and pretraining datasets does not ensure identical checkpoints or implementation details. We will explicitly label the reproduction.

Q5. CKA reference values

We add the requested same-task cross-seed and different-task references:

Comparison of fine-tuning effectsMean CKA
Same task, two native fine-tunings with different seeds (ViT-B/16)0.827
Same task, native source vs. native target (ViT-B/16↔ViT-L/14)0.725
Same task, ARIADNE vs. native target, ext.0.661
Same task, ARIADNE vs. native target, red.0.632
Different tasks, native fine-tunings (ViT-L/14)0.467
Different tasks, native fine-tunings (ViT-B/16)0.426

All use the same validation images. ARIADNE’s CKA exceeds different-task agreement, supporting task-specific structure rather than exact recovery.

Q6. Preliminary cross-family transfer

We transfer from Llama-3.2-1B to Qwen2.5-1.5B. Different tokenizers are aligned by averaging subword activations within common word spans. We use the source base/Instruct pair, updating MLP down-projections of the untouched target base.

DirectionTarget baseARIADNENative target FT
Llama→Qwen29.5139.840.56

Entries are IFEval strict instruction-level accuracy (%). This single-pair word-pooling experiment is preliminary; systematic cross-family evaluation remains future work.

Reviewer_m37tOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ARIADNE, a lightweight method that is able to take a base model, its SFTed counterpart on a task, and another target model (which can have different depth and width, and need not have the same architecture), and uses this to predict a weight update that improves the target model on the SFT task. The way this works is by first defining a mapping from base model layers to target model layers (e.g., layer 1 in the base model corresponds to layer 2 in a model that is twice as deep), collecting representations for the base and target model on a calibration set, computing the Procrustes alignment of both sets of representation (i.e., solving a linear regression problem where the linear map is additionally constrained to be orthogonal). One can then use this alignment as an estimate of what the representations on the calibration set would be for the target model if we were to SFT it, and we assume the alignment to be perfect, then we can cast the problem of finding the weight update for the target model as a linear regression problem.

They evaluate their model and some variants (e.g., a gradient-based variant) to prior work on lightweight adaptation on vision and language tasks, covering model model expansion (i.e., Qwen2.5-0.5B → Qwen-2.5-1.5B) and reduction (vice-versa). They also ablate different components of the model (e.g., using a random/unconstrained alignments). Overall, I think there are some positive results, but there are several shortcomings in the paper that need to be addressed.

Critical strengths

Important problem. In general, devising methods to improve training efficiency is an important problem, and in particular being able to come up with very lightweight methods to leverage training on one class of models to accelerate training on another class of models would itself be helpful for better understanding LLM training dynamics.

Reduction results show promise. Some of the results taking larger base models and their SFTed counterparts yield some results that are arguably positive, i.e., they outperform the smaller base model on the target SFT task.

Critical weaknesses

Ill-motivated applications. While training efficiently (broadly) is an important problem, the specific settings explored in this model are ill-motivated. For example, when considering the problem of taking a small base model and its finetuned counterpart, and applying the ARIADNE-predicted weight update onto a larger model, the smaller SFT’ed model (almost?) always does better. Given this result, I struggle to see a practical application of this technique for adapting larger models, since it seems uniformly better (i.e., less training and inference FLOPs) to simply use the SFT’ed base model. While it is perhaps mildly interesting that the target model is slightly better with ARIADNE adaptation than without, the gap is frequently still fairly large.

Efficiency claims not sufficiently articulated. The central motivation of this method is efficiency, but the experiments do very little in terms of efficiency analysis. For example, there are very large gains in wall-clock time claimed, but there is not enough detail on the particulars of their setup to gauge if this is an implementation/hardware artifact. Furthermore, I think it would be important to also consider other popular mechanisms for lightweight adaptation, e.g., learning a task vector, LoRA, or (if the study were repeated on more recent and larger open-source models; which would also be nice) even just trying to write a good prompt/provide a few in-context examples.

Clarity. In general, I found the paper a bit hard to read at times. Large portions of the text are devoted to fairly simple things (e.g., 3 and 3.1 could probably be much shorter, same thing with Figure 2 and its discussion) but crucial elements of the text are overly terse (e.g., the gradient-based variant is not explained/motivated, the efficiency claims are not elaborated on enough).

Analysis. To me, this method’s performance is tightly intertwined with whether the base and target models can be aligned with a Procrustes map (or, more generally, purely via a linear map). This is a strong assumption, but clearly the results indicate that (to some extent) this works. I feel like the paper could benefit from a better investigation into this phenomenon, trying to quantify the degree to which this is a good assumption and exploring more elaborate (e.g., non-linear) schemes (they might not be as efficient, but could be useful to characterize a performance ceiling of this family of methods).

Questions for authors

Can you explain the gradient-based method? Are the representations used for alignment the gradients for the same calibration set? Is there any principled explanation for this technique (e.g., the non-gradient-based method has a nice motivation)?

Do you have a sense for how these method compare with LoRA, prefix tuning, or even with few-shot prompting?

2 comments

We thank the reviewer for the feedback. We believe the main disagreement stems from a premise we did not state sharply enough, so we clarify it first.

W1. Setting, and why transfer instead of using the source model

Why training-free transfer. ARIADNE addresses reuse of an existing adaptation without iterative target training, following the task-vector transfer and rebasin literature, and the question is how much of an adaptation can be carried over without training. Across architectures, prior methods in this line (Transport and Merge [1], Theseus, BiCo) transport parameters; ARIADNE instead transfers functional effects. Table 1 shows substantially higher accuracy than Theseus and BiCo at 36–39× lower construction time. Key characteristics:

  • No iterative target training. Only forward passes and closed-form solves: no backpropagation, optimizer states, or learning-rate, schedule, and stopping choices; one scalar is selected on validation.
  • No labels for update construction. The update is built in closed form from the two source checkpoints and a small set of unlabeled inputs.
  • Complementary to training. When training is possible, ARIADNE is a better initialization (point 3).

[1] Cui et al. Transport and Merge: Cross-Architecture Merging for Large Language Models. ICML, 2026.

1. The target can outperform the source, even when its post-training cannot be reproduced. The reviewer asks why one should not simply serve the smaller fine-tuned source. Our instruction-following experiments give a direct answer. We employ the official Qwen2.5-Instruct checkpoints, whose heavy post-training relies on large data that are not publicly released: one can download the checkpoints, without access to the complete data and recipe needed to reproduce their original post-training on another backbone. The native target Instruct checkpoint provides a direct reference:

Target baseSource InstructARIADNENative target Instruct
Extension 0.5B→1.5B25.37%31.87% (0.5B)43.52%43.83%
Reduction 1.5B→0.5B19.52%43.83% (1.5B)30.56%31.87%

In extension, ARIADNE recovers 98% of the native gain and surpasses the smaller source, so directly using the source fine-tuned model would be worse here. In reduction, it recovers 89%. For refusal removal, a behavior edit rather than a supervised task, ARIADNE raises HarmBench non-refusal from 27.92% to 67.92% in extension, with a small change on the capability check, while Theseus and BiCo transfer little or none.

2. The target retains broader capabilities. A reason to transfer an adaptation into a larger backbone is to retain capabilities beyond the source task. We test this by comparing the untouched larger target, the ARIADNE-adapted target, and the smaller fine-tuned source on six datasets unused during adaptation: CIFAR-100, STL-10, Flowers-102, Oxford Pets, Food-101, and FER-2013.

ModelAvg zero-shot, 6 unrelated datasets
Larger target base83.65%
Larger target with ARIADNE82.46%
Smaller source FT75.25%

The adapted target is 1.19 points below the larger backbone and 7.21 points above the smaller fine-tuned source. Although using the source retains stronger source-task accuracy, transfer preserves the target’s broader performance while adding task-specific behavior.

3. Transfer helps subsequent adaptation. When further training is feasible, ARIADNE also provides a useful initialization. We apply the same supervised MLP-output-projection tuning procedure to either the target base or the ARIADNE-adapted target, using the same 320 labeled images per task and validation-selected early stopping.

InitializationExtensionReduction
Target base88.26%85.19%
ARIADNE89.56%87.09%

ARIADNE initialization improves subsequent tuning by 1.30 points in extension and 1.90 points in reduction. Transfer is therefore useful both as a standalone adaptation and as an initialization when only limited data are available for additional training.

W2. Efficiency and fairness of the comparison.

ARIADNE, Theseus, and BiCo share framework, hardware (i7-13700, RTX 4080), calibration inputs, and budgets, making the comparison fair.

The phase breakdown is consistent with the methods’ different computational workloads. Per block, ARIADNE solves one Procrustes problem and one ridge regression for a single projection, whereas Theseus and BiCo transport every weight matrix of the task vector through input- and output-side alignments after depth resampling. Table 10 locates the gap in this transformation stage (45s versus 1846–1881s in extension B=10\mathcal B=10); Table 9 reports memory.

These timings cover activation collection, transformation, and update transport. They measure one-time adaptation construction, not inference: the resulting update is absorbed into the native target parameters and adds no inference-time overhead. We will state these boundaries explicitly in the revised paper.

W3/Q1. Gradient-based alignment and presentation

The transferred displacement is Dj=(Sπ(j)ft−Sπ(j)base)QjD_j=(S_{\pi(j)}^{\mathrm{ft}}-S_{\pi(j)}^{\mathrm{base}})Q_j, where QjQ_j expresses the source fine-tuning effect in target coordinates. Default ARIADNE estimates QjQ_j from base activations; the gradient variant instead uses task-loss gradients.

The gradients used for alignment are computed on the same calibration examples and labels in the source and target base models. We compute gradients with respect to paired block outputs and estimate QjQ_j from their centered cross-covariance:

G~s,π(j)⊤G~t,j=UjΣjVj⊤,Qj=UjVj⊤.\widetilde G_{\mathrm{s},\pi(j)}^\top\widetilde G_{\mathrm{t},j}=U_j\Sigma_jV_j^\top,\qquad Q_j=U_jV_j^\top.

The motivation follows BiCo’s interpretation of fine-tuning updates as accumulated activation–gradient interactions. Under a local, one-step approximation, the induced activation change depends on the output gradient. Gradients therefore provide a task-sensitive proxy for the directions underlying the fine-tuning displacement. Aligning them estimates a correspondence between these update-related directions, whereas activation alignment matches the geometry of base representations. This motivates gradient alignment without assuming that the full fine-tuning displacement equals one gradient step.

The larger gains in reduction (Table 8) are consistent with task-sensitive alignment helping select useful directions when projecting the wider source into fewer target dimensions.

In this variant, only the alignment signal changes: the source activation difference and regression procedure remain unchanged; the new QjQ_j changes the displacement’s expression in target coordinates. Unlike the default, this variant requires labels and backward passes, but no iterative target training. We will expand its description and streamline Sections 3–3.1 and the Figure 2 discussion.

W4. Alignment quality and nonlinear alternatives.

ARIADNE does not assume source–target equivalence or exact reconstruction of target fine-tuned activations: transfer depends on how the map transforms the fine-tuning displacement, not only on base-alignment error. Following the reviewer's suggestion, we replaced Procrustes with a two-layer MLP per block, trained to map source-base onto target-base activations. We fit each map on 280 calibration images and report relative error ∥Y−Y^∥F/∥Y∥F\lVert Y-\hat Y\rVert_F/\lVert Y\rVert_F on 40 held-out images, averaged over blocks and tasks.

AlignmentError Ext. ↓Error Red. ↓Acc. Ext.Acc. Red.
Procrustes0.921.3284.38%79.21%
Ridge0.800.7783.15%76.52%
Two-layer MLP0.730.7381.56%76.33%

Better reconstruction of base states does not imply better transfer of the fine-tuning displacement. Procrustes has higher base-reconstruction error yet achieves the highest transfer accuracy, while a random map with the same constraint yields no gain (Table 2). The role of QjQ_j is thus not reconstruction, but a learned, norm-controlled map for the displacement: in expansion it preserves norms and inner products; in reduction it projects without amplifying norms. Unconstrained maps can rescale directions, and a nonlinear map's response to fine-tuning-induced changes is not determined by its fit to base activations.

Q2. Comparison with tuning and prompting.

LoRA (supervised reference). We train LoRA (rank 16) on the same target MLP-output projections, using the same 320 images per task with their labels; learning rate and stopping epoch are selected on validation.

Trains targetLabel-free constructionAcc. Ext.TFLOP Ext.Acc. Red.TFLOP Red.
ARIADNE✗✓84.54%14078.96%132
ARIADNE∇✗✗86.50%20382.29%195
LoRA FT✓✗88.89%378481.49%1072

LoRA finetuning gains 4.35 percentage points in extension and 2.53 points in reduction over ARIADNE, at approximately 27× and 8.1× the construction FLOPs, respectively, while requiring calibration labels and target backpropagation. These results characterize an accuracy–computation trade-off rather than equal-compute performance. With labels, ARIADNE∇ surpasses LoRA in reduction without training the target.

In-context learning. We compare ARIADNE against IFEval prompting with 1, 5, or 10 prompt–response demonstrations. Scores average instruction- and prompt-level strict accuracy (%); parentheses report gains over zero-shot in percentage points.

Zero-shotk=1k=5k=10ARIADNE
Qwen2.5-1.5B25.3731.52 (+6.15)35.37 (+10.00)30.67 (+5.30)43.52 (+18.15)
Qwen2.5-0.5B19.5219.96 (+0.44)21.35 (+1.83)27.05 (+7.53)30.56 (+11.04)

ARIADNE exceeds the best tested ICL setting by 8.15 points in extension and 3.51 points in reduction. More demonstrations do not consistently improve accuracy and require longer inference contexts. ARIADNE instead constructs an update once, with no inference-time overhead to the target backbone.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of rebuttal

We thank the reviewers for their constructive feedback. We summarize below the main points addressed in our responses:

  • clarified the motivation for training-free transfer: scarce data (a small set of unlabeled inputs), low construction cost (closed form, forward passes only), and preservation of the target's general capabilities (m37t, W1);
  • reported native target references, showing that ARIADNE recovers 98% and 89% of the native target Qwen2.5-Instruct gain on IFEval in extension and reduction (m37t, W1; 7oQr, B);
  • added comparisons with head-only, MLP-projection, and full-tower tuning, distillation, LoRA, and few-shot prompting, including matched-compute results (7oQr, A.1 and A.3; m37t, Q2);
  • compared against structured SparseGPT and Wanda compression in reduction (7oQr, A.2);
  • added ViT-B/16 ↔ ViT-H/14 transfer (7.3× gap) across different pre-training corpora, and a preliminary Llama → Qwen cross-family transfer (7oQr, C and Q6);
  • shown that ARIADNE retains the target's broader capabilities on six unrelated datasets and improves subsequent training as an initialization (m37t, W1; 7oQr, W1);
  • compared Procrustes with ridge and nonlinear (MLP) alignment, showing that better reconstruction of base activations does not imply better transfer (m37t, W4);
  • explained and motivated the gradient-based variant (m37t, W3);
  • clarified the timing protocol and reported the cost of multiplier selection for all methods (m37t, W2; 7oQr, Q2);
  • corrected the statement of Eq. (1) for width reduction, where the implemented alignment maximizes cross-covariance; the implementation and results are unchanged (aAMp, W1);
  • measured local and in-network errors after assembling the updates, and compared independent with sequential fitting (aAMp, W2);
  • reported calibration variability, clarified our BiCo reproduction, and added CKA reference values (7oQr, Q1, Q3, Q5).

We will include all these results and clarifications in the revised manuscript. We again thank the reviewers for their time and engagement with our work, and we are happy to answer any further questions.

Dear Reviewers,

As the discussion period approaches its end, we would appreciate your feedback on whether our responses and additional experiments address your concerns. If any concerns remain unresolved or any results need further explanation, please let us know while there is still time to respond. Thank you again for your time and careful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.